Reverse engineering TESLA's software development strategy in production

Last week the second NF-IX Hackathon of the year took place. We drove to Schulenberg, which is located in the mountain region Harz in the north of Germany. Schulenberg is a great venue, offering all what you need for a great hackathon: A great relaxing and inspiring surrounding and a bad mobile network connection, minimizing distraction of social media and business meetings :-)

One outcome of our hackathon was, that looked into the following questions, which have been stated many times by our management: What is so different about Tesla in terms of their development culture? Why does Elon say, they don’t use SAFEScrum? What do the use instead?

So, job understood. A team of people, who did not want to spend their time on hacking, tried to reverse-engineer Tesla based on all pieces of information, which you can find in the web. And that are a lot. We scouted their open positions, containing used technologies, watched plenty of videos on Youtube, and digged deep into web articles. After binge-watching all videos from the Tesla Autonomy days, the answer was clear and so simple: Tesla does a trunk-based software-like development approach in the car production.

This result was amazing but also not surprising. Let me explain you, what it means in detail. Modern tech companies and tech giants like Google, Facebook, Netflix and co emphasize their teams to work on the master branch. By doing this development teams have to merge and release often. By delaying merges the integration pain exponentially increases. So, this is the secret of successful tech companies.

The benefits are manifold. First, you have to build your infrastructure in a way that it support several releases per day. Second, you automatically increase your customer-centricity and customer-understanding, since you release often and are able to test new versions of your code and software features. Tesla does the same in the production. They do not use frameworks like SAFEScrum, where you have to align your development teams. No, instead teams can be created in an ad-hoc manner. The motivation of each and every team is to increase a KPI, which pays off Tesla’s overall vision. So, a KPI can be to increase the customer engagement in the head unit, to decrease the mis-detection rate in the lane detection module of the Autopilot and so on. Super concrete, super mission-driven. The topics in their backlog are prioritized based on these KPIs. Now, a team can set up itself in an ad-hoc manner with the mission to improve this KPI, self-driven, self-organized with the goal to increase this KPI within 3 hours. 3 hours!!! Incredible. If this is not possible, the team dissolves.

The structure is amazing. So, what do you need to achieve such a setup, i.e. a setup which fosters the creation of self-driven self-organizing teams, which are created a decentralized manner but improve an overall vision. For sure, you need a crystal-clear motivating product vision, which can be split down into KPIs. Additionally, if you want to measure your outcome on the KPIs after three hours, you have to either deploy your artefact directly to production in this time span or use good simulations tools. Tesla offers both. Teams are also set up cross-functional with all needed capabilities to bring the artefact to the street. To achieve this, teams can take – also important – decisions on their own. They do not have for the approval of higher managers. Since the KPIs will give feedback, if the result was good or bad, this is obsolete.

So, talking about decision, let’s dig a little bit deeper. Usually decisions are taken by the time. If management is needed, the manager has to decide within 1 hours. In order to have no blockers and leverage the 3-hours-period as efficient as possible, teams can directly reach out to upper and top management.
To sum it up, Tesla’s development approach and mentality is designed for effectiveness. The employee handbook show the same: no long meeting, leave a meeting if you do not contribute, don’t use jargon. All good points in order not to waste time. Tesla’s world is split up into 3-hours-chunks, which are used by tech teams to improve important KPIs.

However, I haven’t answered the question about what I mean with trunk-based software-like development approach in production. So, teams are set up cross-functional and they work near the production line. This means, all software-changes and code, which affects the car, is merged as soon as possible. The production line is more or less the trunk. If a team changes something, all cars are from then on produced with this change. It does not matter, if it is a software or hardware change. For compatibility reasons, software changes are deployed over the air to the entire APIs and API-stability is respected. So, it is possible to bring innovation within days to the car fleet and the entire production line. Elon took the pattern from the trunk-based deployment and applied it to production. Smart move.

After the hackathon is before the next hackathon. Our next hackathon will take place in September, together with CARIAD and the Munich data lab. It will be in Berlin. We’re strongly looking forward to it.