Big data

Showing linear regression coefficients

Posted by

2

I have just finished my Big Data course for 2017, and noted some concepts that I want to teach better next year. One of them is how to interpret and use the coefficient estimates from linear regression. All economists are familiar with dense tables of coefficients and standard errors, but they require experience to read, and are not at all intuitive. Here is a more intuitive and useful way to display the same information. The blue dots show the coefficient estimates, while the lines show +/- 2 standard errors on the coefficients. It’s easy to see that the first two coefficients are “statistically significant at the 5% level”, the third one is not, and so on. More important, the figure gives a clear view

Coef plot from strengejacke Bof the relative importance of different variables in determining the final outcomes.

The heavy lifting for this plot is done by the function sjp.lm from the sjPlot library. The main argument linreg is the standard results of a linear regression model, which is a complex list with all kinds of information buried in it.  Continue reading →

Recent stories on AI, automation, and the future of work

Posted by

0

======================

Melinda Gates and Fei-Fei Li Want to Liberate AI from “Guys With Hoodies”

Who designs software makes a big difference. And Silicon Valley employees are not a cross-section of anything, except each other. Nor need they be; but some balance is needed to make sure products are designed to help diverse people.

As a technologist, I see how AI and the fourth industrial revolution will impact every aspect of people’s lives. If you look at what AI is doing at amazing tech companies like Microsoft, Google, and other companies, it’s increasingly exciting.

But in the meantime, as an educator, as a woman, as a woman of color, as a mother, I’m increasingly worried. AI is about to make the biggest changes to humanity and we’re missing a whole generation of diverse technologists and leaders.  Source.

For one reason this problem is growing right now, see the next story: oligopoly control of AI applications in our lives.

=============================

Another case of the “Big 5” grabbing new AI-related technology before it becomes public.

Apple acquires AI company Lattice Data, a specialist in unstructured ‘dark data’, for $200M

The strength of this pattern, where the Big 5 (Apple, Amazon, Google, Microsoft, Facebook) buy out each novel tech idea and hide it in-house,  as anti-competitive and bad for society as a whole. Apple, because of its level of secrecy, may be worse than some of the others. In a competitive world such purchases would not be a big problem – let the market figure it out. But with the huge cash levels of these companies, which itself indicates monopoly power, they can effectively stifle new ideas that might threaten them in the long run.

  ==========================

Amazon’s new age grocery likely wasn’t technically possible even five years ago.

How Amazon Go (probably) makes “just walk out” groceries a reality | Ars Technica

=========================

Big data and AI are not “objective”

Posted by

0

AI, machine learning, etc only appear to be objective. In reality, they reflect the world view and prejudices of their developers.

 Algorithms have been empowered to make decisions and take actions for the sake of efficiency and speed…. the aura of objectivity and infallibility cultures tend to ascribe to them. . the shortcomings of algorithmic decisionmaking, identifies key themes around the problem of algorithmic errors and bias, and examines some approaches for combating these problems. This report highlights the added risks and complexities inherent in the use of algorithmic … decisionmaking in public policy. The report ends with a survey of approaches for combating these problems.

Source: An Intelligence in Our Image: The Risks of Bias and Errors in Artificial Intelligence | RAND

Standards wars in home automation: don’t spend big $ yet

Posted by

0

19542-285f330fe7918bc83dbe3ba172c81cc2
TL;DR It will take 5+ years for standards to get sorted out in home automation. Until they are, devices from different companies will not be compatible. Anything that you buy and install now will be inconvenient (you will need multiple interfaces) and become obsolete in a few years.

Now that there are many genuinely useful and modestly priced home automation devices (and I don’t mean smart refrigerators), we are ready to enter the rising portion of “the S curve” where penetration increases. Most of the devices can be retrofit, which will make uptake much easier.

But right now, most vendors have their own protocols. Common protocols are needed at 3 layers:  the user interface, such as a mobile phone/computer app (or web site), physical communication such as Bluetooth, Zigbee, or Wi-Fi, and data protocols (API’s, essentially). Most vendors appear to be moving toward a hub and spokes arrangement, where the hub handles communication to the user and outside the home, so there will also be competition for whose hub customers buy. Finally, I would add security as its own “layer,” since it is so important and currently completely neglected.

Continue reading →

“My Galaxy Note7 is still safer than my car.” No, it isn’t.

Posted by

0

The odds of dying in a car wreck are twice as high as this thing “exploding.” I’m keeping it.

Source: My Galaxy Note7 is still safer than my car. I’m keeping it

This author does an interesting calculation, but he does it wrong. The 100 Note7s that have exploded, out of 2.5M sold, were all used for 2 months or less since the phone has only been on the market that long. When you correct for this, the rate of fires over a 2 year ownership period is roughly 1 in 1000. (Probably higher, for several reasons.)

Second, lithium battery fires are nasty, smelly, and dangerous because they can set other things on fire. I speak from personal experience. Do you want to leave a device plugged in at night that may have a .1% chance of burning your house down over the period that you own it? I hope not.

His car wreck odds calculation (1 in 12000), by the way, may be per-year, but again he does not realize that it matters. But he is right that cars are plenty dangerous. I once estimate that at birth an American has a 50% chance of being hospitalized due to a car accident during their lifetime.

There are many other TOM issues to do with this Samsung Note7 recall. Clearly they have internal problems, and problems somewhere in management.

Police  body cams will cost $1000s per cop per year!

Posted by

4

Police body cams sound great, but it will take years to work out all the ramifications, rules for using them, etc. One concern is cost. It’s likely that the initial cost of the cameras is a small fraction of the total cost.

One issue is the cost of storing the video recorded by cams. According to my rough calculations, this could be thousands of dollars per user per year. That will put a hole in any department’s budget.

Continue reading →