The problem I see with it is this: Now, instead of understanding Python, which is straightforward, you have to understand a bunch about Pydantic and type unions. In a large shop of Python programmers, I would expect many would not follow most of this.
Essentially, if this is a feature you must have, Python seems like the wrong language. Maybe if you only need it in spots this makes sense...
I think an important piece of context here is that this is not useful for non-ser/de patterns in Python: if all you have is pure Python types that don't need to cross serialization boundaries, then you can do all of this in pure Python (and refine it with Python's very mature type annotations).
In practice, however, Pydantic is one of the most popular packages/frameworks in Python because people do in fact need this kind of complexity. In particular, it makes wrangling complicated object hierarchies that come from REST APIs much easier/error prone.
> instead of understanding Python, which is straightforward, you have to understand a bunch about Pydantic and type unions.
This like saying "instead of understanding Python, you have to understand a bunch about SQLAlchemy and ORMs" or "instead of understanding Python, you need to understand GRPC and data streaming."
Ultimately every library you add to a project is cognitive overhead. Major frameworks or tools like sqlalchemy, Flask/Django, Pandas, etc. have a lot of cognitive overhead. The engineering decision is whether that cognitive overhead is worth what the library provides.
The measurement of worth is really dependent on your use case. If your use for Python is data scientists doing iterative, interactive work in Jupyter notebooks, Pydantic is probably not worth it. If you're building a robust data pipeline or backend web app with high availability requirements but dealing with suspect data parsing, Pydantic might be worth it.
You're not wrong, but the distinction here that I was responding to was the idea of needing to use Pydantic routinely for typechecking. Libraries that you have to know might as well be language features.
The phrasing of "The engineering decision" in your reply is telling -- you are coming from it as an engineer. But I'm looking at the population of Python programmers, which extends far beyond software engineers. The more such people have to learn, the more problematic the language becomes. Python succeeded despite not being a statically compiled language with clear typechecking because there is an audience for which those aren't the critical factors.
As I said in another response, it reminds me of what happened to Java. Maybe that's just my own quirk, but none of these changes are free.
I think you are underestimating python developers. When python became popular popular languages did not have such expressive type systems. Java and perl were popular then.
Also, here it is claimed the library should be part of the language, and at the same time it us assumed it is too complicated for the users to understand. It seems like the feature being a library solves this, if we let go of the self-imposed requirement of it being part of the language.
When Python started to become popular as Perl alternative, and Zope became a thing to be aware of, I already learned about Caml Light, Objective Caml, Miranda, Standard ML, and the new kind in town Haskell.
Also, even within the constrains of C++98 type system, expressivness wasn't something C++ was lacking.
> When python became popular popular languages did not have such expressive type systems. Java and perl were popular then.
I really like some of the languages you mention, but most of them were not popular, especially in Python domain at the time (scripts and web servers).
The main contenders against Python then were Perl (timtowtdi vs Python one way) and Java.
Java and C++ were strongly typed, but lacking most of the nice things of Haskell etc (at least Java). C++ is very expressive (prob more because of templates than the type system, but I will happily concede this one).
For scripts and web backends, C++ and Haskel/ml were not popular. This leaves Java, perl, php and similar, and at the time neither had advanced type systems in the ergonomic way that it is now expected.
> I think you are underestimating python developers.
There are a lot of people out there writing Python, and a lot of them identify as analysts, (non-software) engineers, scientists, and so on. Not developers. Some of them write immaculate code. Some of them don’t know about git, functions, or commandline arguments, so their code is one long script with big chunks they comment or uncomment depending on what they’re trying to do. The latter are a big constituency for me. Plain type annotations are great in this context, because they place no burden on the user at all. All they have to do is ignore them. Best case, they notice that function f returns a list of floats rather than an ndarray and that saves me having to explain what’s going on.
> Libraries that you have to know might as well be language features.
What you have to know depends on where you're working and what you're doing. You don't have to know GRPC Python libraries, unless it's a company that uses GRPC for internal communication. You don't have to know Flask unless you're building a REST API using Flask. You don't have to know beautifulsoup unless you're building a web scraper. You don't have to know Pydantic unless you're working on a project that uses Pydantic for data validation.
> The phrasing of "The engineering decision" in your reply is telling -- you are coming from it as an engineer. But I'm looking at the population of Python programmers, which extends far beyond software engineers.
You don't have to be a software engineer to make an engineering decision. When a data scientist uses conda because they don't want to manage their Python environment manually, but runs into performance issues on production because their Docker containers are multiple gigabytes larger than they should be -- that's the result of an engineering decision. When a business analyst writes a Python script and manually installs packages without a requirements file, then tries to get it running on a new computer 8 months later but can't because they forget which package versions they used -- that's the result of an engineering decision. So when you deploy your code without any data validation that runs fine now, but breaks in unexpected ways next week because the result of an external REST API you're calling changed unexpectedly...
> The more such people have to learn, the more problematic the language becomes. Python succeeded despite not being a statically compiled language with clear typechecking because there is an audience for which those aren't the critical factors. ... none of these changes are free.
I agree with all this, which is why I said that the engineering decision is deciding whether or not the cost is worth it. Different projects, companies, and people will have different needs.
Your original assertion was that "if this is a feature you must have, Python seems like the wrong language" -- but this contradicts what you're saying. The overhead of learning a single Python library is far, far less than, say, introducing Rust into a company that only uses Python for everything else.
Yes, at that point in time you wouldn't switch from Python. Hence the comments about Java, as an example of where escalating complexity can take you.
I think my assertion, less pithily, was "if having the best type-checking system was critical to you, probably you wouldn't pick Python". And I think that's correct. People pick it for other features.
I didn't say I hated having the option. I expressed reservations, which I still have.
> if having the best type-checking system was critical to you, probably you wouldn't pick Python
Agree, but the only situation where as a developer you can pick a language/ecosystem on its own merits, independently of anything else, is on personal projects. Even if you're a startup CTO building a greenfield app, you have account for hiring and train developers. It's perfectly sensible that you would want to use Python + mypy/pyright/pydantic/etc for extra robustness since it's easy to find Python devs, with a relatively small learning curve if they haven't use those tools, vs going full-on Rust or Haskell, which would require much more rare + expensive people and/or a much longer training period.
> Ultimately every library you add to a project is cognitive overhead. Major frameworks or tools like sqlalchemy, Flask/Django, Pandas, etc. have a lot of cognitive overhead. The engineering decision is whether that cognitive overhead is worth what the library provides.
IMO a library that provides regular functions and values that follow the rules of the language adds zero cognitive overhead. Frameworks that change/break the rules, that let you do things that you can't normally do with regular values, or don't let you do things that you normally could do, are the ones that add overhead, and it sounds like Pydantic is more in that category.
Pydantic is truly a godsend to the Python ecosystem. It is a full implementation of "parse don't validate" and does so using Python's existing type declarations. It uses the same forms as dataclasses, SQLAlchemy, and Django that have been part of Python forever so most Python programmers are familiar with it. And the reason you reach for it is that it eliminates whole classes of errors when the boundary between your program and the outside world is only via .model_validate() and .model_dump(). The outside world including 3rd-party API calls. The data either comes back to you exactly like you expect it to, or it errs. It's hundreds of tests that you simply don't have to write.
In the same way that SQLite bills itself as the better alternative to fopen(), Pydantic is the better alternative to json.loads()/json.dumps().
I don't think you are wrong and I have at times missed having such an option. But... I saw Java go down this path of cool features that you needed to learn, on top of the basic language, and eventually it took Java to an environment where learning the toolset and environment was complex, and vastly changed the calculus of how approachable the language was. In my mind, anyway, it went from being a useful if incomplete tool to being a more complete language that was not really worth messing with unless you were going to make a big commitment.
Every step that takes Python in that direction is a mistake, because if we need to make a huge commitment, Python probably isn't the right language. A large part of the appeal of Python is that it is easy to learn, easy to bring devs up to speed on if they don't know it, easy to debug and understand. That's why people use it despite its performance shortcomings, despite its concurrency issues, etc. (That and the benefit of a large and fairly high quality library.)
I think you're right but I take a different view of it and think it's great. Python is changing so that the language you switch to when you need more performance or type safety is... Python. At some level you have to meet users where they are and large complex applications are already written in Python. And it's those kinds of developers are more invested in the future direction of the language.
I think Python's journey is very similar to Go in this regard where as the language matures and more people start using it for large applications you start having to compromise on the ease of on-boarding in favor of the users who are trying to get work done. Both Python and Go added generics around the same time.
As a matter of fact this would not be a "major undertaking" in Python, unless your definition of the term is majorly loose:
@dataclass
class InnerCircle:
s: str
@dataclass
class NPC:
s: str
@dataclass
class Dissenter:
s: str
Committer = Union[InnerCircle, NPC, Dissenter]
class CocReaction(Enum):
DoNothing = auto()
ThreeMonthsWithoutHumiliation = auto()
PublicDefamation = auto()
def adjudicate(c: Committer) -> CocReaction:
match c:
case InnerCircle():
return CocReaction.DoNothing
case NPC():
return CocReaction.ThreeMonthsWithoutHumiliation
case Dissenter():
return CocReaction.PublicDefamation
Although in reality you'd likely model Committer as a product of a status and name, and adjudicate as a map of status to reaction, unless there are other strong reasons to make Committer a sum.
I think it is normal to know popular libraries of a language. For python, django, drf, fastapi, pydantic or jinja are very common.
There are some people resisting type checks in python but I think fewer and fewer. I dont think people refusing to learn basic concepts and libraries are a reason to not use something.
Also, I am not a big fan of not doing something useful because we need to do a bit of learning. It seems like a variant of "we have always done it this way". Plus it is a strawman attributed to python developers, IMO.
As it just so happens, I was struggling with this in Python recently and this post describes a better solution than what I came up with.
> Essentially, if this is a feature you must have, Python seems like the wrong language.
While I don't disgree in the absolute sense, there are constraints. You can't just switch language or change the problem you're solving. If you have the need for more type safety, then this is a price worth paying.
Okay? Programmers have to understand lots of things that aren't just the bare basics of the language they're using. When did we decide that all software developers are helpless? When can we get back to expecting experts to know things?
When? Probably around the time when the people hoping they don't need to know anything because ai will write what they want discover desire without knowledge doesn't work so well?
Essentially, if this is a feature you must have, Python seems like the wrong language. Maybe if you only need it in spots this makes sense...