Category: Data

Posted on

Week 5: Tiny Oracle

Resources

Overview: Tiny Oracle was an idea I had a few months ago. Its based around aggregating new / traffic / weather / tweets etc into one place, and then generating how a city “feels” based on that data. I didn’t get to the data part this week, but I did make some good progress with the hardware, and even started writing my own restAPI to handle requests. The results, despite it just being a tech test were whimsical.

https://youtu.be/AizbayPvXKM

System: This took a bit of a twisty road. Basically I started thinking I would make a Python based bot that just aggregated all my feed data, parse it, then toss some commands over to a BaaS provider like PubNub or AdafruitIO. But I once again ran into the issue w/ the Feather and its MQTT library, that there’s something buried in it, that prevents it from being totally non-blocking. I knew the particle library didn’t have this issue, so I started digging around in it, and seeing if I could re-mod the mod to re-use for the Feather. But it looked like the particle library was just doing GET requests. So I started looking around for some info on REST APIs, and decided I didn’t need a Pub/Sub setup. I found some tutorials, downloaded Flask (which I like and am familiar with), and starting making a basic API to pass on some JSON. The Feather makes a request on a timer, grabs the JSON the server serves up, and then prints out a message and does a little display based on it. I found it pretty straight forward to build this little Server / Client system. It doesn’t have any security, so its only for local testing, but I would like to make a version that does. But I would like to keep rolling my own solutions going forward.

Components: Adafruit Feather Huzzah, Nano Thermal Printer, FLASK, Python, Adafruit Neopixel Featherwing.

Things I Experimented With: FLASK rest-ful, requests, protocols.

Things I Learned: A lot about basic rest apis! And that you really don’t have to send things to “my butt” if it doesn’t have to go to there. Most of the MQTT libraries for the various BaaS providers seem to have some kind of blocking issue (pubnub / adafruit) in the subscribe function. Also that a lot of data can still just be grabbed with just GET requests.

Future Iterations: I really want to get this to a point where its parsing different data feeds to build its feelings. I also want to keep building a little restAPI for more of my personal projects.

Posted on

Week 4: ServoVane

Resources:

Overview: This week, I wanted to do a project that focused on parsing data out of an API and doing something with it. I decided weather would be the Path Of Least Resistance due to the fact that everyone has a weather project out there somewhere, and the documentation for weather APIs is pretty solid. I also wanted to explore some of Adafruit’s Feather Wings.

System: For this project I wanted to stay away from BaaS providers. You can do a version of this mashing up IFFTT and Adafruit/IO or the like, but I wanted to instead combine all my parsing / data grabbing / behaviour into one board. The Feather is just an ESP2866 on a breakout. Which means it has the ability to do TCP calls, UDP, act as a wee little web server, or just be a gateway. You can drill down into the functionality using various libraries that are compatible with Arduino. For this set up I decided to parse some JSON directly from the Weather Underground API. The servos themselves run on a repeating timer, for a set amount of time. So they are not on constantly. After which they stop, the feather hits a delay mode for a while until grabbing the next weather conditions and triggering the routine all over again.

https://youtu.be/zCgT5Tx5olo

Components: Adafruit Feather Huzzah, PWM Servo FeatherWing, 5v 2a Power Supply, 5V power supply (USB port, but could be a plug in), 7 micro servos

Things I Experimented With: Parsing, Libraries, Mapping, Timing, APIs

Things I learned: There’s some library conflicts with the ESP8266WiFi and Wire.h, in so far that you need to declare the PWM before you start doing any pin setting logic. It could be a timer issue (but the PWM board apparently has its own timer), or it could be a bit of a pin mash. I should have been using a 5v 10a power supply, but I didn’t, so that might have also contributed to some funny behaviour. I also need some extra barrel jack adaptors in various sizes.

Of Note: You need little delays for the ESP2866 to be able to process JSON. If you don’t have them, it can sometimes just not grab things.

Future Iterations: The sounds servos make is pretty nice/annoying. I think working more with the timing and figuring out how to make something like this more of an audio piece might be fun. Working more on system timing and run time.

Posted on

Week 2: Wireshark and Networks

Documentation

Resources

Overview: For this week, my goal was to poke around with Wireshark and learn a bit more about networks. I decided I would also try using pyshark, which is a python wrapper around Wireshark’s command-line utility tshark. Originally I had thought I would pipe some data into Processing to do some viz with it, but at the end of the week realized that Processing already had Carnivore as a library, which allowed packet for in-Porocessing packet sniffing. Wish I’d thought of looking to see if that existed earlier. But oh well? Its nice to know it exists, and I will explore it in the future.

What I ended up doing instead was writing some utility functions to comb through captured data and return the most active of whatever items you send through it. Pyshark has some good ways to dig down into packets, but it can also just grabs all the summaries. So if you want to compare traffic, look for protocols, or just poke around, you can toss together some basic functions to do so.

I think in the end I liked doing the utility functions more than the idea of visualizing it. I would get to the point of having a dataset, and instead of moving on, would go back and dig around in the packets a bit more.

https://youtu.be/yheUVI2p7sM

Of Note: One thing to note though is that LiveCapture in pyshark can sometimes be borked in OSX. Wireshark (and other sniffers) require read access to “/dev/bpf*” which is usually owned by root/wheel. When you use Carnivore, you have to change the permissions on this to 777, which is…very permissive. You will have to do this each time. Wireshark creates a user group, and adds the person who installed wireshark to this group. So pyshark should be able to use tshark under your user without this extra step.

If you can’t get it working then either you need different permissions, or like me, you are trying to read a networking interface that doesn’t exist…because you weren’t paying attention. Then ended up in a 1 day rabbit hole…but I learned some things about what’s depreciated in the OSX system.

Other things I rediscovered: using man pages, and using python’s dir() function. Which is really useful when you want to know what kind of methods you can use on something.

Proudly powered by WordPress