Published Author! Checkout Scaling Java Applications Through Concurrency

I've very excited to announce that my first Pluralsight course has just been published! You can check it out Scaling Java Appliciations Through Concurrency:

https://app.pluralsight.com/library/courses/scaling-java-applications-through-concurrency

If you happen to have a Pluralsight membership, I would love to get your feedback!

Here is the course description from the website:

"There are several gems inside the existing concurrency API that have been hiding in the background for years, waiting to be discovered by curious software engineers. The existing Java Concurrency API makes it much easier to build a Java application that is scalable and performant without having to settle for lots of low-level wait-notify usage or lots of locking using the synchronized keyword. In this course, Scaling Java Applications Through Concurrency, you'll cover several concurrency patterns simplified by the Java Concurrency API; these patterns will make scaling new and existing Java applications simpler than ever. First, you'll learn about how the Java Concurrency API has changed scalability and how to run processes in the background. Next, you'll cover classes that will help you avoid mistakes like lost updates when sharing resources. Finally, you'll discover how to coordinate dependent processes and implementing throttling. By the end of this course, you will be able to easily scale your Java applications through concurrency so that they work better and faster."

I'd like to give a special thanks to Brian Goetz and his book Concurrency In Practice as well as the collective knowledge in online blogs and, yes, StackOverflow. I feel like I learned so much producing the course, and I hope that you get as much out of it as I did.

DVWA 1.9: File Inclusion Medium and High

Although I've studied and practiced secure coding standards for some time now, I had yet to try my hand at the offensive approach before last Friday when I downloaded DVWA and started working on the exercises.

File Inclusion

The file inclusion exercises were unexpectedly eye opening. Initially, I thought: "Directory traversal, get the etc/passwd file, etc., etc., not much here I don't already know." Then, I stumbled into Ashfaq Ansari's walkthrough of File Inclusion and Log Poisoning on DVWA Low which showed to my astonishment how one could use this security hole to poison logs and subsequently upload a php shell to the DVWA server.

Clever. Not bad for a day's work, right?

Medium Level

Thanks to Mr. Ansari, I learned a lot more than I thought I would about the dangers of file inclusion security holes; however, there was more to come. On the medium level, the same directory traversal attack initially seems defended against with the following code:

$file = str_replace( array("http://", "https://"), "", $file);
$file = str_replace( array("../", "..\""), "", $file);

Now, the url parameter value "../../../../../etc/passwd" will instead be transformed into etc/passwd and nothing will show:


Blacklisting is hard, though, and a single-pass search and replace cannot remove all ills. Consider, for example, what would happen when performing a str_replace on "hthttp://tp://". You, of course, would be left with "http://", the thing you were trying to prevent from being in the string in the first place!

So, of course, if all one is going to do is remove the "../" instances from the string, we simply need to construct a string that will leave "../" instances in the wake of a search and replace, e.g. "....//....//....//....//....//etc/passwd" or "..././..././..././..././..././etc/passwd" will both do fine.


Now, the same steps of log poisoning and shell uploading can again be performed with relative ease.

The right way to defend against this is whitelisting, which the higher levels of this exercise employ.

High Level

Actually, I'm not certain quite how to leverage this, yet, but I thought I'd post some of my initial thoughts. The defense against file inclusion in the high level is incomplete because unintended patterns can get passed it:

if ( !fnmatch("file*", $file) || $file != "include.php" ) {
    echo "ERROR: File not found!";
    exit;
}

Here, the regex allows for the file protocol, e.g. page=file:///etc/passwd. Since this would simply serve files from the user's local machine, I'm not sure what could be done with it, but I found it interesting.

A broken CompletableFuture invocation with an unexpected fix

During a recent training, I was demonstrating the class CompletableFuture and how it could be used to create non-blocking methods like this:


    public static void <T> persist(T entity, Consumer<T> andThen) {
        return CompletableFuture.supplyAsync(() -> {
            System.out.println("Persisting entity...");
            return entity;
        }).thenAccept(andThen);
    }

    public static void main(String[] args) {
        persist("toPersist", System.out::println);
        System.out.println("Done!");
    }

I wrote a method like the one above and ran the application. The output, unfortunately, was only:


    Done!

Where did the "Persisting entity..." output go?

(P.S.: The absolute worst feeling ever is when you create an example during a training, and it doesn't work! I definitely should have had a precreated example, but that is beside the point. =])

Since it can be hard to think when there are 30 people watching you, I did the first thing that came to mind. I created an ExecutorService and handed that reference into the thread pool. Now the code says:


    private static final ExecutorService pool = Executors.newCachedThreadPool();

    public void <T> persist(T entity, Consumer<T> andThen) {
        return CompletableFuture.supplyAsync(() -> {
            System.out.println("Persisting entity...");
            return entity;
        }, pool).thenAccept(andThen);
    }

And it worked! It was a bit of a shock, but I was pleased that I could fix it and move on to other topics. :)

After class, though, it gnawed at me, and I was excited when I finally got a minute to sit down and poke at it.

After peeling away a couple of layers, I found that I could reproduce the problem in the following way:


public static void main(String[] args) {
    ExecutorService pool = new ForkJoinPool(1); //Executors.newCachedThreadPool();

    pool.execute(() -> System.out.println("Done"));

    pool.shutdown();
}

This example will only print out "Done" occasionally. However, if I change it to:


public static void main(String[] args) {
    ExecutorService pool = Executors.newCachedThreadPool();

    pool.execute(() -> System.out.println("Done"));

    pool.shutdown();
}

It will work every time.

What's going on? Long story short, I posted my Fork/Join vs ThreadPoolExecutor question to Stack Overflow and John Vint gave the simple answer that all threads in the Fork Join pool are Daemon threads, which means that the VM will stop running without waiting for them to complete. The ThreadPoolExecutor creates non-deamon threads which causes the runtime to wait until they are done.

What does this mean for the in-class example I gave? If you need the runtime to hang on while your CompletableFuture is finishing, pass an Executors-obtained thread pool as a second parameter since by default it uses the Fork/Join common pool.

An Advanced Java Readlng List

Recently, I finished doing a 32-hour training over 8 days for a group for 30 Java professionals. We covered several topics, and I published to them a list of further reading that has influenced me as a developer over the years. Perhaps you will find value in the same list I shared with them:


  1. Head First Design Patterns - My co-workers and I studied this book a chapter a week together over lunch back in 2007. Decorators, Observers, and Strategies completely changed my perspective on how to develop code prepared for change.

  2. Effective Java (2nd Edition) - After having developed in Java for 14 years, I finally picked up Josh Bloch's book and after the first chapter I was sorry I had waited so long. It explained and validated many of my long held practices as well as introduced me to additional good ones, like the self-documenting power of public static final methods as named constructors.

  3. Java SE8 for Programmers (3rd Edition) (Deitel Developer Series) - Java 8 is the coolest Java release since Java 5. Method references and lambdas immediately changed the way I code for the better. In addition, java.time is the Date/Time API we've always wanted. Deitel's book is great at exposing all the great new features that will change the way you code in Java.

  4. Starting Out with Java: Early Objects (4th Edition) (Gaddis Series) - Alternatively, there were some attendees who wanted a good foundational Java book. I've used this book when teaching intro Java classes, and I like it! Especially, I like the early introduction to object-oriented programming.

  5. Java Concurrency in Practice - This is another eye-opening book that I waited far too long to read. Atomicity I understood, but I had never considered the ideas of visibility or re-ordering. And I was completely helped out by his explanation for what to do with InterruptedException.

  6. Java Performance: The Definitive Guide

  7. Iron-Clad Java: Building Secure Web Applications - I included this one because security is often tacked on after running some vulnerability assessment. While I might agree that one should "optimize after", security should be built in from the start.

  8. Mastering JavaServer Faces 2.2 - To be honest, I prefer an action-based framework like Spring MVC, but JSF 2.2 caught my eye, especially with the new HTML5-friendly JSF attribute syntax. More than that, this group of engineers asked for training on it, and this is a great book to get a more comprehensive view. :)

  9. REST in Practice: Hypermedia and Systems Architecture - This book is not quite as practical as the rest of them, but I really liked the theory outlined here, mixed with code examples. Through this book, I better understood the gaps that exist between existing libraries and what ReST specifies.

  10. Pro JPA 2: Mastering the Java(TM) Persistence API (Expert's Voice in Java Technology)

  11. Java Message Service - This book helped me understand an API that had to that point had seemed so inaccessible to me. That and Spring Boot made it super-easy! :)

Enjoy!

Lemon Squash and 20 Minutes of Coding


HTML-encoded Lemon Squash
A couple of weeks ago, I decided I would try an experiment with my two oldest children and my wife sort of similar to the time I planted a Lemon Squash in our backyard:

Have we tried it before? No.
Do we know if we'll like it? No.
Do we know if it will even grow in our climate zone? No.

Sounds like a winner!

So hear goes... having kids learn along with non-coder Mom is like growing lemon squash in the garden:

Lemon Squash is Hearty


We learned very quickly that lemon squash didn't require a lot of maintenance, wasn't vulnerable to squash bugs like all our other squash plants, and basically grew even with us often forgetting to water it. For a family of six kids, that's a big plus. :)

Likewise, having the boys learn HTML using codecademy.com was a very self-directed process. Compared to the logical dead ends that my students at Neumont will run into with if statements and for loops, my boys, in the world of HTML, were relatively impervious to bugs. The closest they got to a head scratcher was the following:


  ...
  <h3>Seven Things I Like To Do
  <p>Play the piano</p>
  <p>Read books</p>
  ...

(This was made trickier because codecademy said that the solution was correct.) In HTML, all tags must be closed when we are done with them, much like turning off the bathroom light when you are done with it (this analogy worked for my older, more rules-conscious son). When the above is rendered by a browser, all three lines are in bold:

Seven Things I Like To Do
Play the piano
Read books


 Of course, my boys don't know that this was a bad thing and neither does my wife. They didn't know what it is supposed to look like in the end.

However, when my wife checked it over, she noticed the error based on the detailed instructions and explained the concept as she understood it to the boys. The boys fixed the error, and it then looked like this:


Seven Things I Like To Do

Play the piano
Read books

At that point, my wife said "Oh, so that's why it was all bold: The browser thought the paragraphs were part of the header." Bingo!

Small bugs like these quickly became easy for my boys and my wife to squash (see what I did there?) repeatedly until the entire species retreated into extinction.

Further, other than tag issues, my boys needed very little direction from Mom. Without any parental instruction, my younger son began shouting "DOCTYPE!"
at the beginning of each exercise since he knew it was required no matter what, which filled me with the same amount of pride as the first time he recited the "Inigo Montoya" line from the Princess Bride by heart.

In the end, they were able to get through the first 15 exercises with only minimal correction from my wife. I personally never once corrected the kids nor my wife. Sometimes my wife would ask me a theoretical question or two after the kids went to bed, though. (By the way, those conversations where fun. It was the first time that I can remember my wife showing a technical interest in what I do for a living.)

While hearty, the process wasn't perfect. There was some churn around the fact that codecademy would often tell the boys they were right when actually there were some (what I would consider) important syntax bugs in the HTML they produced. Perhaps the software was built to be lenient. That said, my wife caught the issues and was cognitive enough to know the difference between following the instructions and just getting the software to say "correct!".

Lemon Squash requires water, soil, and sunlight just like every other plant


My wife and I, it turns out, didn't really need to know much about photosynthesis to effectively grow good lemon squash. We just needed to do simple, understandable things like plop the plant in soil that gets sun and pour some water on it. Still, those were necessary elements of making it grow.

Likewise, my wife didn't have to have a lot of formal training, be a genius, or do anything more than use the skills she already had to be there for her boys.

You don't need to be a genius to help kids learn to code.
To be truthful, my wife is a smart cookie, so I won't diminish how her university education may have been brought to bear so that she could assist our boys with such ease. The truth is, though, that codecademy did most of the work. They would read and follow the instructions and the software would do an okay job of guiding them when they misunderstood and did the wrong thing. Thereafter, my wife would "water" by checking things over and helping them make the corrections that the software incorrectly skipped over.

The flow that worked the best was to water regularly. Kristi would hang out
You could do this, right?
while the boys completed the exercises, and she would check each exercise the moment the boys finished. This was a bit of an investment in the beginning; however, after the boys understood the start-and-end-tag thing and a little bit about nesting tags, they were on their way. The first couple of times, Kristi did not do this, resolving to check all their work in the end. This ended up taking much longer because mistakes made on the first exercise would perpetuate through the next five or six. This way, they would practice the wrong way for 20 minutes and then need to be retaught by my wife (which required her to go back through each exercise on her own, doubling the time it took to make progress). This created frustration for all three. Once they changed to the more hygienic practice of checking after each one, it only took a minute or two cumulatively; they boys made more progress and Kristi was less frazzled.

Lemon Squash doesn't taste good to everyone


In the end, I was the only one who would eat the Lemon Squash. I never really understood that because I thought they were excellent. Of course, my children "prefer" only that which is suger-laced, so that could be part of it.

I asked my boys what they thought, and they said it was "fun".
<p>I like trains.<p>
They liked putting silly phrases in the header and paragraph tags and making lists of hobbies, TV shows they liked, and actors they found annoying. They liked adding images to the page and linking to their favorite sites.

I asked my wife and she said, "No, I'm just too busy." For her, squeezing out 5 20-minute sessions where she directed the learning environment was tantamount to me asking her to pull all our children from the public school system and prepare them for college by herself. It made me admire her support all the more, but it also helped me to see that not every parent will be able to carve 20 minutes consistently out of their day to sit and code with their child.

Conclusions


I'm interested enough in this that I'm going to try and get a few more non-coder parents to try it out. The coding world is less scary than it used to be and non-coders can teach other non-coders to code using tools like codecademy. Anyone interested?

It was important for an older person with more training on paying attention to detail to be there with my boys to make sure they stayed on target With a small initial investment, they became quickly self-directing for 90% of the time. It would be interesting to see a button like "Have a Live Volunteer Look At Your Code And Give You Tips" on codecademy or some other site so that Mom or Dad can have a bit more flexibility.

There were a couple of times when the software was incapable of noticing that the syntax was incorrect. Software has bugs, and that's okay; a way to still be helpful while bugs are being fixed might be to have an example picture of what the solution ought to look like in the end so students have a visual way to spot check their solution against the instructions.

Twenty minutes a day was good, though I think that it would eventually need to turn into more. The exercises are written to allow someone to learn piecemeal, but as the concepts get trickier and more abstract, it may take an hour or two of practice before a student feels confident in her understanding. This kind of time commitment would likely be when the student really decided to invest himself in non-trivial coding (like going from picture books to chapter books when learning to read).

Non-coder parents should try this out! My wife did it, and so can you!

When Strings Are Not Immutable In Java

Give this code shot and see what happens:


public class MutableStrings {
    private static void toUpperCase(String str) {
        try {
            Field f = String.class.getDeclaredField("value");
            f.setAccessible(true);
            f.set(str, str.toUpperCase().toCharArray());
        } catch ( Exception e ) {
            // yes, I'm eating an exception! (cuz the above turns out to be a *horrible* idea, so why not compound it with another bad practice?)
        }
    }

    public static void main(String... args) {
        final String greeting = "Howdy";
        toUpperCase(greeting);
        System.out.println(greeting);
        System.out.println("Howdy");
    }
}

You should get the very unexpected output of:


HOWDY
HOWDY

What just happened??


So, I was having fun the other day with reflection in one of the Java classes I teach at Neumont University. Based on some cool stuff I learned when I got GSSP certified two years ago, I proposed that strings could possibly be mutable if Java was run without the security manager, making setAccessible(true) available. Sounds like it's worth a try:






public final class MutableStrings {
    public static void toUpperCase(String str) {
        try {
            Field f = String.class.getDeclaredField("value");
            f.setAccessible(true);
            f.set(str, str.toUpperCase().toCharArray());
        } catch ( Exception e ) {
            // yes, I'm eating an exception! (cuz the above turns out to be a *horrible* idea, so why not compound it with another bad practice?)
        }
    }
}

With this handy method, I can now pretend that strings are, indeed mutable.

Instead of


public final class Mainer {
    @Test
    public void testStringToUpperCase() {
        String greeting = "Hello";
        greeting = greeting.toUpperCase();
        Assert.assertEquals("HELLO", greeting);
    }
}


Now, I can do


    @Test
    public void testStringToUpperCase() {
        String greeting = "Hello";
        MutableStrings.toUpperCase(greeting); // notice the lack of assignment on the left
        Assert.assertEquals("HELLO", greeting);
    }
}

But, WAIT, there's more!


Because Java uses a String pool, two separate references set to identical string literals can point to the same reference, e.g.


    @Test
    public void testStringPoolEquals() {
        String greeting = "Hello";
        String salutation = "Hello";
        Assert.assertTrue(greeting == salutation);
    }
}

Because greeting and salutation point to the same reference in memory, if I change the value property in String reflectively, I get the following behavior:


    @Test
    public void testStringToUpperCaseStringPoolCorruption() {
        String greeting = "Hello";
        MutableStrings.toUpperCase(greeting);
        String salutation = "Hello";
        Assert.assertEquals("HELLO", greeting); // okay, makes sense
        Assert.assertEquals("HELLO", salutation); // wait... WHAT??
    }

Woah, what just happened?? Why did salutation get changed as well? Well, now we've come full circle: Strings are pooled in memory in Java. When I do "String variableName = string-literal;", it will search the string pool for that string and fashion a reference pointer to that object in memory instead of creating a new one. When we edit the internal value directly, everyone that points to that same object will get the modified value instead.

Less effective...


Obviously, this is a terrible idea. :) Imagine the chaos of something like string comparison failing because someone reflectively changed the underlying char[] array of a string-pooled public static final variable. It also seems like it could be exploited somehow, tricking an application into rendering malicious information to an end user, like in the case where a hacker gets arbitrary code to run on the server.

Still, it's fun to amaze your friends!

"Have You Done Your 20 Minutes of Reading...er...Coding today?"

When I first became a parent, I was filled with the excitement of being able to teach my kid everything that I knew. I'd teach him how to be creative, responsible, curious, empathetic (at least as far as I understood them), but I'd also teach him fun things like the first 314 digits of pi, juggling, basketball, and coding.

Yes, coding. As a software engineer, I never worked a day in my life. I *love* to code. It's fun. I'd do it even if I weren't paid to do it. And actually, now that I'm a teacher full-time, I can make good on that claim. Although the majority of my day is now spent teaching, I still squeeze in time to code, and it is largely for the fun of it.

To my coding friends: Do you remember the thrill that came from the first time you got your computer to do something you programmed it to do, even if it was simple? I personally remember getting my first Pascal book in high school, and I literally could not put it down. One particular memory I still have from my 14-year-old self is sitting in the back seat in the McDonald's drive thru, nose stuck in my Pascal book while my sisters talked about Jonathan Taylor Thomas or some such*.

I wanted my kid to have the same thrilling experience, the same overwhelming passion. I wanted to see the deep focus come over them that comes over me when I have a terribly abstract and difficult problem to reason around and the subsequent exultation when they finally see the light at the end of the tunnel and realize that they did it.**

For me, teaching my kids coding was always a want. I heard about Scratch, and I downloaded it. I learned about Greenfoot and downloaded it. I tried them both on my kids with a mild amount of success. I spent an afternoon with my 8 and 10 year old boys and built an animation where three stick people would run up a ladder, slide down a slide, and fly into the wall, including sound effects that would only attract 8 and 10 year old boys. Another afternoon, we built an LDS missionary app where the missionary had to walk around and visit all the houses before the "bees" caught him. On another day, we built a simple frogger game. Another, we built a "Family Home Evening Spinner". We had fun.

Now, though, I believe I'm transitioning. "I want" is slowly turning into "I should" or "they need".

Reading Coding is Fundamental


I think about coding like I think about reading and writing. Research, studies, and our own intuitions seem to confirm that reading aloud to our children every day enstills within them a skill that will assist them in solving big problems, getting a larger world view, and generally being productive in society. Clearly, not being able to read and write is an enormous disadvantage in this day and age.

In parts of the world where literacy is low, those who are literate are invaluable to their community, often serving in positions of leadership and working at the forefront of the community's biggest problems.

Sound like what coders are doing today? One need only look as far as heavily code-literate institutions like Google to see some of the impressive problems that are now being solved today that were once considered intractable or at least only part of the distant horizon.

As Gabe Newell from Valve put it: "The programmers of tomorrow are the wizards of the future. You're going to look like you have magic powers compared to everybody else."

I'll take that a step further, though. I don't only want my kids to have those "magic powers" in order to make big, positive differences in the world. They are going to need them.

Douglas Rushkoff explained that "When we acquired language, we didn't just learn how to listen, but also how to speak. When we acquired text, we didn't learn just how to read, but how to write. Now that we have computers, we are learning to use them but not how to *program* them."

Largely, the world seems satisfied to allow themselves to become increasingly dependent on "geeks" for increasingly fundamental aspects of their day to day lives. Many not only consider the inner workings of computers and the Internet "unknowable" but even "worthless to know".

However, I believe that such an imbalance between the knows and the know-nots, while convenient in the short term, will put those who don't know how to code at as much of a debilitating disadvantage in the future as not knowing how to read does today. Those who could write centuries ago formed the world in the image they saw as multitudes of non-readers consumed their perception of reality, often independent of its accuracy or precision.

For me, teaching my kids coding was always a want. Now, though, "I want" is turning into "I should" or "they need".

Our children are the consumers of the digital revolution that many of them cannot currently participate it at a production level. Their perceptions even now are being molded by the coders behind Facebook, Google, Instagram, and Twitter. If our youth do not know how to code themselves, they are destined to be relegated to the class of citizens who consume points of view formulated not necessarily by those who are right but by those who are simply more skilled at the lingua franca.

As Patrick Byrne from Overstock once quipped: "Mark Twain once said don't pick a fight with someone who buys ink by the barrel. I say never pick a fight with someone who buys bandwidth by the gigabyte." Kids needed to learn to write. Kids now need to learn to code, too, if they are going to have a say in the makeup of our future world.

"Have Your Done Your 20 Minutes of Coding Today?"


For some of my kids, they love reading so much that they don't need to be reminded. One of my kids reads at the dinner table, in closets, and with a flashlight late into the night. Another reads because it is on her "job chart" and she wants to put down the check mark. A couple of them need to be reminded over and over again if I want to have any kind of output from them.

Regardless, they all know that Mom and Dad expect them to read 20 minutes a day. We read them bedtime stories, we read the Bible and Book of Mormon every night together as a family. They know that Mom and Dad love to read and like to carve out time for our own personal reading during our day.

Kids need daily time coding as well, and it needs to be on the job chart. Or, if that seems too daunting, families can try the rule that we enforce in our house:
When on the computer, be a producer, not a consumer.
My kids are allowed to build stuff in Minecraft, draw pictures in paint, make sound effects and animations in Scratch, learn something new by researching online or anything that basically is not straight up media consumption. We have "Cummings Cash", and I tie that Cummings Cash to certain bounties that are related to them building, creating, and making stuff on or off the computer.

It's time for the Cummings's to level up, though. With our oldest two children (8 and 10), I want them to get familiar with coding more than just as a pastime. So, for one week, I decided to try something new.

The regimen was simple: Work on codeacademy, a codemonkey level, or something coding-related for 20 minutes a day together for one week. I considered the pairing up part to be super-important as my older son is more responsible, but my younger son has more of the "knack". Hopefully together they'd stay on task.

In addition, it's also been on my mind to see whether or not a non-coder parent can help a non-coder kid become more familiar with coding. I imagine that mother or father 300 years ago who had never learned how to write, but now perceived the need for her children to learn to write. What would she do in such a situation to help her children make progress?

So, while not the greatest comparision, I decided that as an additional experiment, I'd ask my wife--a non-coder--to sit in with the boys for those 20 minutes a day. She would follow along with the exercises and help the boys out if they got stuck. We called it the "Code With Your Child" initiative.

To my delight, my wife accepted the challenge! This week, she is working through the exercises with the boys; sometimes she sits and watches, but mostly the boys work on the tasks independently and she verifies after they are done. It's going well! My boys are learning coding from a non-coder! I will post a detailed update when our experiment finishes next week.


In the mean time...


Why don't you give it a shot yourself? IMHO, everyone should know at least a little bit about coding. And maybe pull your kid in with you while you do it.

*Actually, I'm sure my sisters were more sophisticated than that. Kimmie would accept nothing less then Jason Priestly or that one guy from New Kids On The Block.

**My wife tells me that she knows it was a really hard problem when I yell out a "Whoop!" when it finally works.