A new profession takes a time to truly develop, if it's animal husbandry, barbering, bridge building or what have you. For software development, the journey is just underway.
The software profession took clear steps forward during the 1960s. Programmers had become essential to U.S. military defense, and, led by IBM, it was entrenched in American industry
The imperative to go to the Moon set a lot of money to circulating, and there was interest and enough of it to fund study of how software teams were successfully formed, and how their results could become reliably reproducible.
If you had the right methodology, your process could be repeatable, said the expert, just before telling you to that you could throw away yours and use his. that his methodology was the best. Predictable.
There was Yourdon and Booch and Rumbaugh and Popkin and so on.
That is overstated for effect, of course. So is the joke: Q: What is a methodology? A: It’s a method that went to college.
The great clarion of software engineering was sounded by Frederick Brooks, in a paper that was to become The Mythical Man-Month. Working at IBM, Brooks studied and found some surprising truths about complex software and projects, the most telling being that where he proves that projects actually slow down more when leaders begin adding more people to a project as it gets closer to actual completion. Brooks and others gave a philosophical underpinning for structured analysis.
One of the methodologists, Grady Booch, traces advances in software engineering to the SAGE project, the Cold War computer network. SAGE stood for semi-automatic ground environment, and it ran from the late 1950s to the 1980s. Iit presented a unified image garnered from many radar sites, and was inspiration for one of the satirical props of the "war room" in Kubrick's With the purpose to defend the country from nuclear attack, it cost more than the Manhattan project.
In the SAGE era there was a tremendous amount of software written [ 22:38 ]
I am afraid our leader is not
particularly stable. - Capt. Mandrake
Out of defense systems' experience arose Structured Analysis, or Structured Methods. And that was where waterfall methods. "You analyze your requirements, you design everything up front, and then you start building things," as Booch puts it. This approach grew a mind of its own, and eventually ran its course. But it served a purpose or two.
The advent of the software engineer meant there was someone you could put in jail, or at least discharge, if their software killed someone. When there was need for new title for raises at the turn of the centrury, arose then did the software architect (Paging Mr. Art Vanderlay!)]
Now, with the data scientist, we take another step on the winding road to professionalism. Are they really scientists? Of course not. And what do they need more than anything? Good judgement - of the kind that comes from experience, and humanistic understanding of the context of data.