Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Learn more about Collectives

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Learn more about Teams

I have a neural network trained in pytorch that I'd like to deploy into a Unity app. What's the best way to do it? I'm also interested in allowing the user to further train the neural network in the Unity app, which I guess would require to integrate some part of pytorch into Unity (maybe there's a way to integrate pytorch's C++ / torchscript API with Unity?). If anybody has experience with this, I'd like to know what the best alternatives are.

unity3d.com/machine-learning Unity has a machine learning toolkit you may wish to check out. – Doh09 May 26, 2019 at 20:59 I'm familiar with ML agents, but this doesn't answer my question. ML agents is just a collection of environments that have to communicate with an external ML framework. I'm interested in embedding a ML engine into a Unity app. – Mei Zhang May 27, 2019 at 0:52 Never claimed it would answer your question. Mentioned it in case you were unaware of its existence. Hence I made a comment not an answer. – Doh09 May 27, 2019 at 5:11

Check out the new features in Unity ML Agents. There is an inference engine within Unity ML Agents (called Barracuda) that allows you to use pretrained models within your app. AFAIK, you can convert Tensorflow and ONNX models into Barracuda. It should not be a problem as Pytorch models can be converted to the ONNX format. You may need to retrain your model if it is directly affected by the app (for example, if it is an RL agent).

EDIT: To answer your second question, you can continue to train the model but not in real time. What you may be able to do is collect data from the user, and use that to further train the model (that is how TensorFlow Serving works). You can do that by converting the PyTorch model into a TensorFlow model via ONNX.

EDIT 2: Barracuda is now a standalone and production ready inference engine that runs exclusively on the ONNX format. Any framework that can be converted into the format (e.g. Keras, Pytorch, MXNet) will work as long as they contain the supported operators.

Could you please elaborate on what are the supported operators and how to implement them? – Logic1 Aug 27, 2022 at 6:32

This is not a complete answer. But it should be able to help you progress further.

Essentially you'd as I see it just need to be able to run Python code within C# code. In this case in collaboration with Unity's framework.

I did some searching and came across 4 partial solutions:

Unity python interpreter: https://forum.unity.com/threads/python-interpreter-in-unity.86461/

An example of running code with IronPython in Unity: https://gamedev.stackexchange.com/questions/123526/a-python-script-controlling-a-unity-game

Communication example between Unity3D C# and Python, using ZeroMQ: https://unitylist.com/p/hc8/Unity3D-Python-Communication

Implementing language support for other languages yourself via an XML file: https://forum.unity.com/threads/add-multiple-language-support-to-your-unity-projects.206271/

Once you're able to run the code you'd then need to refer to the location of your python files. This way you should be able to run it if the environment you run it in also has Python installed and set up correctly.

You must ensure that the files are in your project on deployment. And that Unity can access them.

Hope this helps you.

Have you been able to train mlagents within a build using one of these methods? Also, does this mean there is no way to somehow ship python within the build, so that the end user isn't required to install python on his computer? I am trying to create a distributable application in which the user is able to train his own agents. Currently I am facing the python roadblock and I am considering fully implementing my RL algorithm of choice in C#... – Florian Wolf Jul 7, 2020 at 18:46 While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review – miken32 Nov 30, 2021 at 17:29

Thanks for contributing an answer to Stack Overflow!

• Please be sure to answer the question . Provide details and share your research!

But avoid …

• Asking for help, clarification, or responding to other answers.
• Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers .