Category Archives: posts

Full text posts

US Data Investments to Support the Next Generation of Ag Innovation

While we all know about the US government’s big investments in agriculture research, their role as a big data and analytics provider is less well appreciated.

Seeing the phrase ‘big data and analytics’ we think of machine learning, artificial intelligence, cloud computing, and other popu;ar 21st century concepts, but the US government has been providing important data and analytics to researchers, farmers and their advisors, and ag tech companies for over a century. The US has been the innovation leader of global agriculture for most of this period, and weather and soil data, nutrient models, geospatial information and crop yield data have all beem important components.

Soils data is a great example, from formal soil surveys starting in 1899, to today’s USDA soils resources wealth of data and models. If you want to dive in, check out the amazing SSURGO dataset, which you can access (in a kind of clunky way) through the Web Soil Survey.

These data and analytics assets are an amazing resource and a huge, national competitive advantage. Importantly, they are all available free of charge to anyone, which is not the case in many other places around the world.

The US strategy for innovation-enabling ag data and analytics warrants a longer discussion, but let me kick it off with a list of 5 opportunities for the the USDA to have an even bigger impact:

  1. Farm Efficiency. An array of new practices have emerged since the turn of the century, including new tillage strategies and cover crops. On the surface these seem sensible, but do they achieve their goals? Is there a solid cost-benefit win, or are they a mixed bag? Helping to inform grower decisions with real data and analysis has been a key role of the USDA, and it is an ideal time for a deeper look at the impact of these emerging practices.

  2. Soils, Topography. The US soils data is great, and it is a critical tool for managing water usage and resources. Small changes in the sand content of soil or shifts in the shape of the terrain can make dramatic changes to how much irrigation is required in a given season. As we need to manage water more carefully in the future, any improvement in this data, especially in high irrigation regions, will be invaluable.

  3. Weather. Weather data is the single most important factor in nearly every aspect of crop modeling, including fertility, irrigation, yield, disease and pests. The US gets big points for making all weather data free for all uses, but there are opportunities to improve the spatial density of precipitation-related forecasts and readings, and to improve inputs and calculation of key derived values, such as evapotranspiration.

  4. Satellite Imagery. I predict we will look back on the Twenty-Teens as the start of the golden age of satellite data in ag. Fly-bys will increase from every 10+ days to every day (cloud cover permitting), and our ability to leverage what we’re seeing will increase radically with more data. Most of these new capabilities are private (namely PlanetLabs), which is , but there is an important role for the government to play in providing freely available data of increasing quality and scope.

  5. Genetic Data. When we think of agricultural genetics we naturally think of GMO corn, but that’s a tiny fraction of what will matter in the future. Other crops, the soil microbiome, and the genetics of disease and pests will all be impactful areas for research and applications. Historically the big ag companies have taken the lead in genetics, but between with their capture of intellectual property and international ownership, there is a strong argument that the USDA should secure the core genetics data to enable a broader base of research and private sector innovation.


Data is increasingly viewed as an asset, and that has been true in agriculture for decades. The US government deserves credit for recognizing this before most, and now its time to look forward, continue to invest, and modernize their approach to match today’s technology and priorities.

Who Owns Tomorrow’s Autonomous City Cars?

Jump forward a bunch of years. Cities continue to get bigger and even more people need to get around. Uber/Lyft-like services have evolved and are more efficient and better than ever. Full autonomy has also been realized – there are no human drivers in the city. Autonomous vehicles of all sizes are dashing around and providing awesome, cost-effective, on-demand transportation throughout the city.

Question: who owns all of these autonomous vehicles?


  1. Random people. They “rent” them out to on-demand services when they’re not in use. Seems super unlikely – why bear the cost and hassle of owning a car in the city when such great service exists?

  2. The Uber/Lyft companies of the day. Maybe? A very different business model than Uber and Lyft today, and, again, kind of a hassle to own those distributed assets.

  3. Car manufacturing companies. Also maybe. Again, pretty radical business model shift from today.

  4. Cities. The new model beats mass transit, so they outlaw private vehicles and run the fleet themselves. My bet is some city will do this.

  5. Specialized fleet management companies. Similar business models exist today, such as car rental companies, but do they transition?

I’m actually not sure – any thoughts?

The Internet-Enabled Eclipse

We saw the 2017 eclipse in Oregon. Here’s some quick thoughts, and why I think this one was a singular event.

  • I feel lucky on two accounts: 1) having the opportunity to be somewhere to see it with my family, and 2) having good weather. It would be easy to miss it for either of these reasons.
  • The eclipse was very cool, but totality was a whole different level of cool. (There have been lots of good write-ups, but I thought Jason Snell captured it well). The combination of the corona, being able to look at the sun directly (and also see Venus!), and the level of darkness were all a bigger deal then I would have guessed. If you have the opportunity to get to totality in 2024 then definitely do it!
  • The goofy eclipse glasses were totally worth it.
  • The change in temperature was noticeable where we were. Cliff Mass summarized the weather impact in the northwest.. Interesting that some of the stuff was a surprise…..
  • Cliff also mentioned the traffic. In general there were some delays, but quick to get through. HOWEVER, the state of Washington once again proved that they have the doubler whammy of bad roads and bad drivers. Serious traffic jams on I-5 in the middle of nowhere,10 hours after the eclipse and 100 miles north of totality.

This eclipse was “marketed” as the “Great American Eclipse”, but I think what really made it unique was that it was the first total US eclipse of the Internet age. Everything from websites to social media to camera phones to e-commerce played a role in this eclipse. Location selection, travel plans, weather, traffic and sharing – all of these involved the Internet. In short, the 2017 eclipse wouldn’t have been itself prior to the Internet.

Eclipse today: As we were driving away from totality my teenage daughter said “cool, one of my friends saw the eclipse from a plane and just posted the video!” (I don’t have a link to it, but here’s a similar one).

1970’s eclipses: it was interesting for me to go back through the historical list of US total eclipses. I remember being outside my grade school in Wisconsin with the paper-and-pinhole setup viewing an eclipse. From the timeline, I suspect it was in ’70 or ’72. The February of ’79 eclipse happened while I was in high school, and I can attest to the fact that it was a non-event (I actually don’t remember it at all).

In the 1970’s, how would you have known where to travel to? What time the eclipse would peak at that location? Would this info have come from a school? 6 o’clock news? Newspaper? How would you even have known to get excited about it? How would you have gotten eclipse glasses, let alone known about them? How would you have shared your experience? Would you have visited the AAA store to get a Trip-Tik so you had some maps?

As things increasingly happen at “Internet speed”, it gets harder and harder to remember what things were like before it existed. The 2017 eclipse provides a one-time opportunity to contrast the the world before and after the Internet.

This entry was posted in posts on by .

Remembering Jim Canty


“… a first ballot Hall of Famer on the ice and in life.”

The quote above was from a mutual friend in reference to Jim Canty, who passed away in late April. If you’re racking your brain for memories of Jim’s professional career (“Wasn’t he on the ’84 Nordiques?”) you can stop now. Jim’s illustrious 22-year career was with Hippy Hockey, the Sunday night skate at the local rink.

It would only be partially accurate to characterize Hippy Hockey as a bunch of old guys reliving their glory days on the ice — many of us never had glory days to relive. Jim loved hockey. Like myself Jim came to Hippy Hockey via pond hockey, and enjoyed the magic of skating, the friendly competition and a beer or two in the parking lot afterward. In the summer Jim would bring fresh clams from the Cape and cook them up.

Jim and I initially connected through our kids. My two older kids were similar ages to the middle two of Jim’s four kids, and intersected in everything from play dates to confirmation to hockey games. We coached hockey together one year when the boys were young.

Jim was, at his inner core, a family man. I’d seen Jim in his husband/father role, but his family says it started early on with his parents and 8 brothers and sisters. When Jim and I saw each other we always caught up on how each other’s kids were doing. When some people talk about their kids and their accomplishments it comes across as bragging, partly about the kids, but often more about how great of a parent they are. With Jim it was different. He spoke with a sense of selfless wonderment and joy. I wish I could describe it better, but if you’d had the chance to talk to Jim you’d get it.

Professionally Jim was an lawyer and investor, so there was little overlap with my tech world. We did, however, interact about some of the energy investments he was looking at. I’d try to help out with some technical perspective, or tap into some of the science talent at work to analyze some startup’s claims. Over time I came to realize that Jim and I shared a similar optimism about human potential. Sure there’s lots of problems in the world, but there’s also a lot of smart, hard-working people, and humanity has the potential to overcome its challenges.

I wasn’t the only one who noticed this about Jim. At his memorial service a Franciscan monk made a connection between Jim’s optimism and the optimism at the core of Franciscan values. Jim had strong connections to the Franciscans, a Catholic religious order who follow the teachings of St. Francis of Assisi, with roots back to Jim’s college days at St. Bonaventure.

I’ll close with a statement from the Siena College mission/vision page, since it describes so well for me how I saw Jim live his life:

In our Franciscan community, optimism is a faith-filled affirmation of the basic goodness of life and of all men and women because, in the words of St. Francis, God our Creator is “good, all good, supremely good.” So:

  • be positive

  • be hopeful

  • be open to the future.

May God bless Jim and his family.

This entry was posted in posts on by .

Remembering Ken Traub

Just over a week ago Ken Traub, my friend and colleague, suddenly passed away. Lots of people have shared their memories of Ken, and have captured what a wonderful person he was, his intellect, and his breadth of interests and skills. To say that Ken was exceptional was an understatement, but I wanted to highlight three specific aspects of Ken that stand out for me.

The first was Ken’s ability to organize complex very systems. To the average person this might not sound that “sexy”, and they’d be mostly right, but the foundations of our digital world depend on a small number of people with this unique ability.

An example is the system of numbers that show up in bar codes and RFID tags and are a foundation for our entire system of commerce. Over the years Ken made major contributions to these standards and technology. Creating a system like this that can work reliability on a global scale requires the synthesis of information from a wide range of topics, including physics (the ability to read a code accurately), to computer science, to business standards and processes. Knowledge of all of these need to be synthesized into a coherent design, and then combined with the ability to write down the design in a clear and complete matter.

This last point is particularly important. A key artifact of these projects are lengthy, detailed, precise documents that specify everything about the system. Most of us (and I definitely include myself) will get the basics right, then lose momentum when it comes time to work through all of the edge cases and exceptions that occur in the real world.

Successful projects in this space produce systems, like the bar code and RFID systems, that are so good, so reliable, that you forget they exist. Ken was among the world’s elite at creating and defining systems like this. (Side note: the other person who was amazing at this that I’ve had the privilege of working with was Guy Steele, who also happens to live in the same town as Ken’s family).

The second thing I’ll always remember about Ken was his intellectual honesty. When you design things in groups, lots of interesting dynamics appear. Individuals might get attached to their own idea and defend it even after it has been shown to be flawed. Or engineers defend idea A because it links to idea B, which they are really excited about. Or someone dismisses key information because it comes from someone they don’t consider to be as smart as they are. Or you’ll get competitive types who view the design process as a contest where there are winners and losers depending on who’s ideas get built.

These behaviors are so common, so widespread, that over time you listen to people with the assumption that there are hidden agendas at work. Quickly I learned this was not the case with Ken. Whether we agreed or disagreed, he was always driven towards an underlying beauty or truth, the belief that there is a “right” answer to any design challenge, and a willingness to incorporate new ideas or information that might help lead to that result. It was fanstastically refreshing to design things with Ken – you just knew you were going to end up with a better design, and the journey was going to be as rewarding as the destination.

Finally, my lasting image of Ken is mentoring a junior engineer. The act of mentoring embodies so much about Ken’s character: his modesty, his approachability, his teaching skills and his amazing thought process. Ken didn’t just help other engineers solve problems – he taught them to be better engineers.

And that was Ken in a nutshell. You thought you were sitting down to improve a design, but you got improved in the process.

God bless Ken, and give his family strength during this challenging time.

This entry was posted in posts on by .

More on United…

Following up on the post from my good friend Snowman on Fire, I wanted to add a few notes. Like Snowman I have been a customer of United and Continental for decades, with over 2M miles. Long ago I came to a healthy point in my relationship with them – I understand when I take them I’m risking random crap like this happening, so except for extreme cases, I don’t let it bother me. I only fly them when they’ve made it well worth my while through a much better fare, travel times, upgrades, etc. All airlines have their warts and good sides, but its not a fluke that this happened on a United flight instead of someone else.

  • Snowman is on the money with his points: United was within their legal rights, but the situation was self-inflicted (caused by their own operational ineptitude), and of course they handled it all wrong.

  • Companies can develop personalities, and United is somewhat schizophrenic. They can have good days, but they also have really bad days where they are downright mean spirited. I believe them when they say they didn’t know what the security people would do to the gentleman, but I have no doubt that many of them were hoping that is exactly what would happen.

  • Much of the bad side of their personality comes from the pre-merger United organization. If you have a good flight crew on United and ask them which company they came from, 8 out of 10 times they’ll be from Continental.

  • The fact that they kicked a passenger off for an employee is totally within their culture, which is “employees come first”. United flight crews tend to cut in TSA lines without a “sorry” or “thank you” more often than the others, get their bags on board first before the overheads start to fill up (look for them next time you board), and heaven help you if you try to get some service while they are figuring out their schedule for next month.

  • I had something this bad happen on a United flight a good while back, before cell phones could capture and spread it. Tried to get United’s attention for two weeks after, but no one cared (and I know I was not alone in calling people there up – the main conversation on the flight was “did we really just see that?”).

Finally, I want to add one more point to Snowman’s observation that this was self-inflicted, because United had created the situation. Not only did they create the operational situation, but if the passenger was a regular United customer, they were in no mood to do any favors for the airline.

This entry was posted in posts on by .

AI, Society and Citizen Engineers

Two recent Techcrunch articles highlight some of the challenges we’re in for with the increased use of machine learning (ML) and artificial intelligence (AI), to the extent these are separate. In [AI’s open source model is closed, inadequate, and outdated], Kumar Srivastana argues that we need a new kind of transparency (I’m avoiding his use of “open source” – more on that below) because of the complexity and unpredictability of these systems. Along similar lines, Jeremy Elman and Abel Castilla argue that we need to rethink liability and quality standards in [Artificial intelligence and the law].

These articles cover some interesting ground, but are also enlightening in the misconceptions that they reflect. Some specific points:

  1. “Open source” can technically mean one of two things: 1) an organization’s decision to make code or other intellectual property visible to others, and 2) a set of licenses organizations that allow to control how others can use material they choose to make visible. I agree with Kumar that AI and ML introduce some new elements that organizations can choose to make visible, but it’s not clear that the decision process of organizations or the licenses we use are the issue.

  2. Even without AI or ML we have many systems that people depend on every day where we have no idea how they work. Furthermore, these systems are complex enough that no one truly understands how they will react in different situations (e.g. cascading failures in our electric grid), and we have no visibility into how these systems are being tested, etc. AI and ML may further complicate this problem, but to portray the situation as brand new and unlike what came before it is not accurate.

  3. Both articles discuss AI’s as somehow disembodied beings. These are algorithms embodied in products and services, where they are generally replacing large, complex pieces of human-written software. Maybe I’m missing something, but I don’t understand how the replacement of one algorithm with another changes the liability of the companies for their products and services. If my product uses software and it fails in a damaging way, AFAIK the liability should be the same independent of how that software was created.

For me the bottom line is this: AI and ML are subtle and potentially powerful tools. As with any technology, it is the responsibility of product and service companies, and, importantly, their engineers, to understand these tools and use them in a responsible way. And since these are rapidly evolving technologies, it is incumbent on engineers who use them invest in the time to stay current and keep their systems up to date with the latest methods for testing and validating the algorithms they produce.

Elman and Castilla provide the excellent example of a traffic light that is run by an AI, and the AI decides that the most efficient mode requires the lights to change faster than normal, resulting in more accidents. The authors cite this example as an example of why we need the law to adapt. I disagree – this is a clearcut example of engineering culpability. Just because you use AI or ML for part of an algorithm doesn’t exclude you from putting in some good old-fashioned logic checks in addition. Think about it: if a human were controlling the light, wouldn’t we want some logic to make sure they stay within some safety parameters?

Artificial intelligence and machine learning are important new technologies, and they bring some interesting twists. But this is the next in a very long history of technologies, going back to fire if not earlier, where it is the responsibility of our engineers to translate them into safe uses. I know that it’s scary that we can’t look at a piece of code and see exactly what it’s supposed to do, but its foolhardy to believe that with today’s complex systems anyone fully understands what they do and can vouch for them anyway.
As far as I can tell our existing legal and transparency frameworks and practices haven’t been shown to be outdated by these new technologies, so let’s see how their use evolves and react when our systems break down.

This entry was posted in posts on by .

Sustainability and Values, Birds and Wind

I have always maintained that every action justified by sustainability has side effects, and that it becomes a values-based judgement as to whether the value of the sustainability improvement outweighs negative impacts of the side effects. Being a values-based judgement means that not everyone will view the cost-benefit tradeoff the same way. Further complicating matters is the fact that the tradeoffs are apples versus oranges, as in the case of compact florescent light bulbs, that have much better energy efficiency than our historical incandescent bulbs, but also have embedded mercury that poses other threats if not handled appropriately.

One such cost-benefit tradeoff that deserves far more attention is the threat of wind power to birds and bats. The fact that giant wind turbines kill birds isn’t a surprise, but we (the US public) don’t know how many. The problem is that the data of has been kept secret, though its value is questionable since collecting the data has not been a requirement for wind energy providers. However, we can get a hint from US Government decisions, and what is revealed is not pretty.

On the 14th of December the Department of the Interior (DOI) finalized a rule that absolves wind generation companies from penalties and prosecution for killing protected birds for the next 30 years (news coverage by AP here), overriding the Bald and Golden Eagle Protection Act that us normal Americans are governed by. The rule allows up to 4,200 deaths per year of bald eagles, or roughly 3% of the total population. It also waives penalties for killing the rarer golden eagles.

While the number of eagles killed each year is kept secret by the DOI, we have some insight into the number, since this DOI rule raised the cap on bald eagles by nearly 4x, indicating that the wind turbine owners were nervous that they were at or above the current limits.

Personally I’m appalled by the death of these beautiful birds. However, more appalling are the lack of good, public data on bird deaths by wind farms, and the lack of an open discussion about the relative value of wind power and birds. How many dead bald eagles per year are an “acceptable” side effect of our growing wind power? How many golden eagles? The DOI has decided that this is a discussion they are not going to allow to happen.

Why We Want Biased News

All of the recent discussion of news, be it fake, unfair, slanted, etc, has gotten me thinking about the mechanics of news. I’ll define news as things or events that, as far as anyone can tell, happened or didn’t happen, and reported newsas how news is communicated, including websites, papers, newscasts on TV, radio, podcasts, blogs, tweets, etc. So any description of something that happened or didn’t happen, then I’m calling it reported news.

First, I assert that all reported news is filtered, meaning that it’s not the whole story. This is true at every level: any news show or website or paper cannot describe everything that happened that day, and for any individual story, be it a political speech or a car accident or a good deed, it’s impossible to provide all of the detail and background. In every case what’s reported is a small subset of what is being reported on, so is filtered.

Second, I assert that all useful filters are biased, i.e. they favor some things over others. There’s almost always a spatial (favor local news) and temporal (favor recent news); we’re so used to these we don’t even notice them. The bias can also favor one group over the other in sports, religion, race, politics, sexuality, age, eduction, etc, etc, etc. And the bias can be in attitude, favoring happy, sad, good, bad, etc.

Even though news has always been biased, our awareness of the bias seems to be going up. This would make sense – we have more news sources, many of which are targeting specific audiences, and aren’t even claiming to be unbiased. As a result, it is easier to see and compare the choices that specific news outlets are making.

The New York Times has had the motto “All the news that’s fit to print” since 1897, but in 2016 no one treats that motto as an absolute statement, i.e. that the New York Time has some magic filter that isn’t biased. Seeing that motto today people naturally read “All the news that’s fit to print, as determined by some folks at the New York Times”.

Techies may tell you that algorithms can be unbiased, but that’s generally not true. The hand-coded algorithm will embody the bias of its developer, and the machine learning algorithm will learn the biases of its trainers. Beyond that, there’s the question of what inputs the algorithm has – does it have access to all of the details of everything that’s going on, or just a subset? So sensing system of our algorithm can be biased as well.

The clever coder might now ask: what if I had cameras everywhere, and created an algorithm that randomly creates a stream of news from all of the world’s events? My response is that it would be super boring: minor traffic accidents in far off countries, youth soccer scores for children we don’t know, misbehaving mayors of cities we’ve never heard of, etc . While our natural inclination is to think of bias as bad, the truth is that we want bias. We want an editor, human or algorithm, to find things that will interest us.

The key, then, is to embrace the bias, and acknowledge and understand it. The only time bias is really bad is when pretend that it isn’t there..

This entry was posted in posts and tagged on by .

Self-Driving Cars: Advantage Pedestrians

In Boston, two warring factions have been partnered in an unlikely, decades-long truce, their aggressive tendencies controlled by an unspoken, yet universally understood set of rules of engagement. That truce may soon be shattered.

Boston pedestrians will brazenly cross busy streets in the middle of a block, but deep down they understand that getting hit by a car could really suck. Boston drivers are aggressive, using intimidation to navigate rotaries and keep pedestrians in line. But they also understand that hitting a pedestrian would be a major hassle.

The result of this mutual understanding might look horribly chaotic and dangerous to an outsider, but it generally works, making near-optimal use of the road for both groups. Of course a mini van with Ohio plates might get stuck for a few minutes by some commuters coming out of a subway stop. Or a foreign family may wait 3 or 4 light cycles before summoning the nerve to trust the “walk” signal, but that’s the price of stepping into the middle of a conflict without understanding the rules of engagement.

Now picture this scene from a not-too-distant future: a mutual fund manager wants to grab a Starbucks before digging into company valuations for the morning. Before stepping off the curb mid-block, she looks down the street to assess the state of oncoming traffic, and the only car with a real shot at her has no driver – a freshly programmed autonomous vehicle. Instantly she recognizes that lawyers in some glass office building have ensured that the AI in that car will do absolutely anything to avoid hitting her, so she confidently steps off the curb.

As more autonomous vehicles hit the streets, this becomes scene occurs more and more often. Self-driving cars take an hour to go two blocks on Boylston St during commute time. On Saturday afternoon on Newbury St, fancy electric cars are stalled until their batteries run out, blocking traffic for hours afterwards.

It’s not clear what the answer to this emerging problem is. Rehire all of the out-of-work cab drivers to hand out $500 jay walking tickets? Blanket the city with advanced facial recognition cameras that try to shame repeat offenders on popular social networks? Underground roads? Overhead walkways?

It’s possible that nothing will work until we’re willing to make the artificial intelligent cars just a little big crazy.

This entry was posted in posts on by .