Unity, C#, Game Development
Writing a Unity Editor Script
Learning how to write editor scripts in Unity
When trying to learn something new it is usually a good idea to come up with a small project that uses whatever you want to learn. Or at least it seems to work very well for me. A few months ago I accidentally discovered that you can write editor scripts in unity and then I found a reason to write one.
I bought the forest environment package when it was on sale on the Asset Store and wanted to do something with it. But somehow I just can’t get familiar with how editing terrain works in Unity. After a while I just gave up and cried in the corner. Well I didn’t give up completely. After cleaning my tears of disappointment I saw it as an opportunity to learn more about editor scripts and share what I have learned.
Table of Content
- 0. The Main Idea
- 1. Making a simple Editor Script
- 1.1 Creating a custom Inspector
- 1.2 Event Handling
- 1.3 Drawing Gizmos
- 2. Editing the Polygon
- 3. The Ray Casting Algorithm
- 4. Spawning the Objects
- 5. Final Touches and Thoughts
0. The Main Idea
The biggest problem I have when I try to create a terrain in unity is, that it is often super slow. Also I can’t figure out how you can place multiple trees at the same time.
What I would like to have is a simple tool that allows me to define an area where assets are placed. I imagined that a nice way to do so would be to just define a polygon somewhere on the terrain, specify the density of the objects and then place everything by pressing a button.
1. Making a simple Editor Script
To make the editor script we first make a normal unity script, a C# class that extends the MonoBehaviour class.
To make the actual script we make a new class that extends the Editor class and has a [CustomEditor(typeof(SomeScript))] annotation.
If we now select a GameObject that has our Unity script attached to it we would see the results of the editor script in the Unity editor. Sadly there is nothing to see yet. To get our editor script to do something we will implement three main functions:
- OnEnable
Here we are going to initialise everything in the editor script. - OnSceneGUI
This is where we can do the event handling, things like mouse button presses or key strokes. Another thing we will do is draw gizmos, in our case the polygon where objects will be placed. - OnInspectorGUI
Here we will put all our buttons and sliders or rather everything that we would like to see in our custom inspector.
1.1 Creating a custom Inspector
The custom inspector that we are going to create will be fairly simple. All the methods that we need are static and can be found either in the GUILayout or the EditorGUILayout class. The methods in GUILayout can be used ingame as well and EditorGUILayout works only in the editor.
The methods that we are going to use are the following:
- GUILayout.Label()
Renders a label, helps to make things more manageable - GUILayout.Button()
In case we want a button :3 - EditorGUILayout.Slider()
Creates a slider, we will use it to set things like randomness and density - EditorGUILayout.Toggle()
Creates a checkbox - GUILayout.BeginHorizontal()
This is used to render widgets next to each other - GUILayout.EndHorizontal()
Put this where you want to stop rendering widgets next to each other - EditorGUILayout.ObjectField()
Here we can load prefabs from out project
1.2 Event Handling
The event handling needs to go into the OnSceneGUI method. We can access the current event with Event.current. If it was a key that was pressed then Event.current.type would be equal to EventType.KeyDown and to get the KeyCode we can access Event.current.keyCode.
1.3 Drawing Gizmos
Gizmos can be drawn using the Handles class. We are going to use:
- Handles.DrawAAPolyLine
Draws lines specified by the points of an array. We use this to draw the edges of the polygon - Handles.DrawSolidDisc
Draws a solid disc. We will use it to draw the vertices of the polygon - Handles.PositionHandle
This is used to draw a position handle. It is the same thing that pops up when you select a GameObject and press W in the editor. We will use it to move the vertices of the polygon around.
2. Editing the Polygon
There are two or three things we will have to do inside the editor script. We have to somehow set the vertices of the polygon when we click somewhere. That should be achievable by using the physics API in Unity. We can use HandleUtility.GUIPointToWorldRay to create a Ray from the camera to where our mouse position is in the GUI. Then by using Physics.Raycast we can the get the corresponding point on the terrain. The following code segments uses the mouse cursor to shoot rays.
Then we will have to somehow spawn objects inside of the polygon. What I thought about doing is to project the polygon into the 2D plane and create a grid of points for which I will then check whether they are inside the polygon or not.
After that when I want to spawn an object I will again use ray-casting, to cast a ray through the point on the grid and check where it hits the terrain. There I will then spawn the object.
Here is the updated version of the editor script. We can now add and delete vertices and move them around with the position handle. By pressing PageUp and PageDown we can chose which vertex is selected currently (might be cooler to do this in a different way). Also, adding new vertices is disabled if we are orbiting around the selected GameObject (Event.current.alt == true) or if we are moving the position handle (GUI.changed == true).
3. Ray Casting Algorithm
How do we know whether a point on the grid is inside the polygon or not? There are many ways to do this, but I decided to use the ray casting algorithm. A point is inside a polygon if a ray starting from said point going in any direction intersects with the polygon an uneven number of times.
Remark: There are some edge cases, like it our point lies on a vertex or an edge of the polygon, but they are irrelevant for what we are doing, so I am going to ignore them.
To use the ray casting algorithm we have to first find out whether two lines are intersecting. We can do that by using the amazing power of linear algebra (oh no… not the scary mathematics :S)
Assuming that two lines are defined by
l₁ = {p₁ + v₁ ⋅ t, t ∈ [0, 1]}
l₂ = {p₂ +v₂ ⋅ s, s ∈ [0, 1]}
Then they are intersecting if there exists t ∈ [0, 1] and s ∈ [0, 1] such that
p₁ + v₁ ⋅ t = p₂ +v₂ ⋅ s
After some juggling we get that
(-v₁, v₂)⁻¹ ⋅ (p₁ - p₂) = ( t, s)ᵀ
Where (-v₁, v₂)⁻ ¹ is the inverse of a 2x2 matrix with columns -v₁ and v₂. So we just have to compute the above and check whether t ∈ [0, 1] and s∈ [0, 1]. Now we can either implement this ourselves or if you are lazy like me you can just use Math.NET Numerics. Just download the dependencies and put them under the packages folder in the Unity project.
Here is the code that I used in the end:
Remark: It might be that it is possible that you can just use the physics API from Unity to do this, but I didn’t have a good idea how.
All that is left is to generate the ray for a given point and count the number of intersections with the polygon and check whether it is even or not.
4. Spawning the Objects
All that is left to do now is to spawn the objects and attach them to the terrain. To get an object into our editor script we use EditorGUILayout.ObjectField. This allows us to select a prefab from our assets. Then we will generate a grid inside the polygon that we will then use as the basis for where the objects will be spawned. What the following script does, is generate a square shaped grid around our polygon and then decides for each point, whether it is inside of it. Since it is mainly regular C# I will not go into the details.
Now that we have the points we have to add some randomness to them and figure out where to spawn the objects. For that we will generate Rays that originate roughly 100 units above the point downwards and check where they intersect with the terrain.
Now that we know where to spawn the objects, we have to instantiate them and attach them as children to our terrain. You can just use the normal Instantiate method, there is nothing wrong with it. I am not going to do that. I will use PrefabUtility.InstantiatePrefab. The reason why I do this, is because in a later version of the script I want to figure out what prefab has instantiated the spawned object. That information gets lost in the Instantiate method. When we instantiate the object, we will also give it a random rotation to let it look a bit more natural.
Remark: If you ever want to get the prefab that was used to instantiate the object you can use PrefabUtility.GetCorrespondingObjectFromSource.
Here are the results of this script
5. Final Touches and Thoughts
5.1 Final Touches
The script so far does work, but it is not that pleasant to use. In the final version I added some quality of life improvements.
I allowed the user to create multiple layers where objects can be spawned.
Also, instead of just spawning one objects I added the functionality that allows to add as many objects as the user wants and then spawn one of them randomly.
Then, a problem that the script as described above has, is that the polygon would not follow the terrain if the terrain is moved around. I fixed that in the final version but left it as it is in the tutorial for readability.
Another thing that I added is that when we select a GameObject with our script on it the first time, we will load all the layers and prefabs again. That way it is a lot smoother to use.
5.2 Final Thoughts
There are still some things I would like to add to the script but I leave it as is for now.
It would be nice to get some visual feedback for the density. I first tried to draw the points where things would be placed, but I ran into performance problems with the way I was drawing the points and couldn’t figure out a nice way to do this without performance issues in the editor.
It would also be nice if the Objects would fade out as they get closer to the edges of the polygon. This could be implemented and is not exceptionally complicated, but as of now I am too lazy to do so. I would have to use high school math… so boring :D
When spawning a lot of objects the script sadly takes quite a while. I am not so sure how to fix that. Maybe I use some stupid things in my code. Also a progress bar would be really nice. That way the user of the script would at least know that Unity didn’t crash.
All in all I learned a lot when I wrote this script. I hope that it was understandable what my thought progress was and how I worked.
If you have any questions or suggestions please feel free to contact me.
The final version can be found on my GitHub.
