Saturday 31 October 2009

SenseCam - a new type of video compression








Some SenseCams I built




image on right shows SenseBulb messages


There have been quite a few articles re Sensecam in press recently re Microsoft licensing it. It is more than just a time lapse camera. I designed it initially in 1999 as a new type of video compression system to capture all the interesting moments in life - but leave out the repetition and dull bits. Many of us sit behind PCs all day and there are better ways of capturing the pc desktop see Microsoft's MyLifeBits as an example. (MyLifeBits does as a whole lot more, but this is one interesting feature of it, here is the book) Patent introduction above shows my 2004  invention of Sensecam for Alzheimer's.
In SenseCam, sensors such as heat sensors, to detect people, light sensors, to trigger a sequence of pictures of rooms as person walks through doorways. An accelerometer can detect user moving and capture image and also reduce blur in the image. A microphone is triggered on loud sounds and images captured. Tagging data with GPS as in my SmartQuill patent here. Even a heartrate increase can be logged with SenseCam images. These events may happen about once every 20 seconds and played back as a continuous sequence of images at about 5 frames per second. That gives an approximate video compression ratio of up to approx 1:100.
Here is a 2007 video example of my bike ride, the compression is around 1:15 here as quite a lot of action events when cycling. It is from Girton, Cambridge, up Huntingdon Road, past the Astronomy Labs to Microsoft Research.
A less dynamic sort of day will allow compression of around 1:100 or better. My "normal" days are probably a bit like many of us engineers , behind a PC, writing and debugging software, hardware debugging with oscilloscope and soldering iron. (I ignore email if I am debugging software as interrupts train of thought).
The SenseCam also captures around 400% more of the view as it's close to a fish eye lens view 104 degrees compared to around 50 degrees on normal camera.

JPEG comparison is a compression ratio of approx 1:100.

What next after SenseCam? After I left Microsoft I set up Girton Labs and designed SenseBulb technology to capture all that the Sensecam missed. It is a camera with heat imaging and captures life in all it's minutiae. It's first application is a camera that replaces a lightbulb and sends mobile phone text messages of activity in the house. Thermal images are replicated as "colour shadows" and so no real privacy issues. Mundane events such as back doors left open, fridges being opened, people moving around the home, eating, sleeping, TVs switched on indicate the "aliveness" of a house and it occupants. This is useful for Assisted Living, Alzheimer's and people living on their own. A family member will get a text message saying fridge door opened 3 times this hour, that means all is well.


I guess some people would find it useful to text or Twitter "I'm home" completely automatically via SenseBulb rather than manual typing. SenseBulb transmits encrypted text messages messages via GSM network so as not to inform burglars, Twitter is for all to see. I have prototyped Sensebulb V1 and its fun.
A bit more here on SenseBulb. (Patent filed) Page very out of date as current version V2 with colour pixel display now being tested.

Email me if any questions on this technology at sensecam@gmail.com or call me on +44 (0) 7970 101578 (Cambridge UK).




Girton Labs's new sensing camera



Friday 30 October 2009

SenseCam Cosmetics

The SenseCam case has had small changes  since my first minimal design of 1999 then 2005, case molding by Microsoft Hardware Group.
Why so minimal? I design all my computers with no on/off switch, I use sensors like accelerometers or light to power on/off design.
From left to right, you can see the 104 degrees wide angle lens, enough to capture image of your shoes, LED status indicators, Passive Infra Red Sensor (PIR) and light sensor. The bump is for the AA cells, a lot easier to find in a hurry than a Lithium charger. I am interesting to note that the new Sensecam still uses a huge filter for the PIR.
The lanyard was nice and simple - until someone decided to "upgrade" it to magnetic catches using neodymium magnets. I had to quickly email one of our principle Microsoft Researcher users of sensecam, who had a heart pacemaker, to not be alarmed. Neo magnets are 5 times more powerful than normal magnets, and used to control the settings of pacemakers!





Friday 23 October 2009

an early sensecam 1976




I started experimenting with sensecams when I was 17. I had left school, and this was a paying vacation job between school and going to university. My other paying jobs were writing for Practical Electronics and Electronics Today International. The company was Baxall Electronics, Bramhall, Stockport, Cheshire, and consisted of me the first employee and Malcolm Baxter the boss, and owner Don Hall . The company was based in a small wooden shed. Malcolm had invented "Watchman" a Eurocard size board full of TTL and analogue components that could be used to set a zone on a video image to send alerts. Typically a control knob would be used to set a maker over a door in the view, and so video recordings of intruders could be captured. I did the testing of these and other video devices. Vidicon tubes keep burning up. I confess I never did tell Malcolm it was my error. I learned a lot about signal to noise ratio in analogue video circuits, great fun. Malcolm was ingenious, the video signal and power went down same bit of cable. My job was to optimise the transistor circuit for the best signal to noise ratio. Baxall Electronics grew to become one of the largest supplier of CCTV in the UK and the world, with several hundred employees but was taken over in 2007. Cameras were initially put in the London Underground. 33 years later the UK now has more cameras in public than any other country in the world. You might be viewed up to 300 times per day on London Cameras.
Roll on Microsoft's Sensecam.


Wednesday 21 October 2009

Why no audio on the SenseCam?


Many people ask why there is no audio on the Sensecam when I invented it for Microsoft?
It can be done, one, it is easy to use a standard digital audio recorder (e.g. maybe the new Apple ipod nano) to record audio with time synchronised information to sensecam video. Two, my concerns over privacy issues, conversation is likely to be inhibited if you know audio is being recorded. Some people just dry up and you never hear their best (or worst) jokes! Third, I did design an audio compression algorithm for Sensecam. This uses snippets of sound bytes from the day, e.g. train sound, London Underground, car engine sounds, laughter, footsteps, kettle sound etc to add an atmosphere of sound, and adventure to the video. It did not record conversation. The video stills are playing back at about 100 times faster than the days' events so the audio needs to keep up. I was developing this when I was deleted from Microsoft, and the work never got patented. It was based on detecting sounds over a preset amplitude and an FFT filter to pick out interesting sounds of artefacts. There was a microphone in early version of sensecams, see it in image, by the strap.

The iPod nano allows users to capture audio as here The Nano also captures video as well.
It is interesting to note synchronised music can be added to SenseCam videos, here are some.
First ever sensecam video here and another here

Disclaimer, I am a Microsoft Shareholder
I work part time for Apple Inc as Consulting expert
on my patent sold to them re iPhone sensor technology as here.



Lyndsay Williams
www.girtonlabs.com
sensecam@gmail.com


Friday 16 October 2009

A brief history of Microsoft's Sensecam





Oct 15th 2009, Microsoft finally license Sensecam see here. This is a sensing camera to help people with Alzheimer's.
(Image shows early version of SenseCam)

I invented and built the first sensecam in 1999, my patent filed 2004 as here , first paragraph explains my plans for helping Alzheimer's .



 I had been approached by Roger Needham, then head of Microsoft Research Cambridge in 1998, to work for Microsoft Research after invention of my SmartQuill, a sensor telephone appeared on www.slashdot.org , this technology now used in iPhone. * See end of article.
Why did I invent SenseCam?
I wanted to help a person with memory problems who was always losing keys, also a dentist friend of mine, Dr Jane E. Jacott, Woodford,  Cheshire,  wanted some technology for time and motion studies of how dentists spend their time, sitting, standing, doing surgery, paperwork, etc. This really inspired me and concerned me for some months, as I wanted to help her.  I also have lost a few months memory after being hit by a car . It is torture to lose your memory after a car crash. I think it was so much worse for Mum & Dad as they knew all but I lost my memories for about a year.
I built the first SenseCam using an Analog Devices accelerometer, a PIC microcontroller a digital camera that could take 64 images, a good capacity for 1999! It was hand soldered on Veroboard and programmed in PIC assembler by me in 4KB of memory. I attached it to front of my bike basket to capture images of all the drivers in Cambridge who kept hitting me. I curiously ended up with more pictures of ladies not looking after children in buggies and buggies ran in road and I got pics of wide eyed scared babies when I braked ! (accelerometer captured braking and triggered pic) But a lot of the images were poor and blurred and pic of a car bumper is dull. I searched for months for a suitable tiny fish eye lens and found one , this allowed a very wide angle (104 degrees) image. So wide angle it could even remember the shoes I wore on the day! I put the project on hold for a year or so until I found a camera module with 128Mbyte of memory. Meanwhile I was working with John Krumm, of Microsoft Research Redmond on people location badges using radio and accelerometers. This worked like indoor GPS. I have to say at Microsoft Research, my employer, there were many other supportive folk, but Andrew Blake was one of the the best. (Roger Needham also but sadly died,) Andrew knew I had being doing experiments with accelerometers on the body and prompted me to do a device with lots of raw sensor data output so the machine learning people could analyze it. Perhaps add a camera? The key bit to getting sensecam working was my algorithm for recording a sequence of rooms. I said to my then husband, who kept loosing his keys, which room where you in? Library, car, hall, lounge..retrace your steps. I designed software and hardware with a light and colour detector, accelerometer,temperature sensor and passive infrared detector. When we walk through a door, a step change in light occurs, i.e. a door frame detector, so an image can be triggered. Person in front of you, triggered by heat, captured image. Accelerometer, used to capture pic when you turned around, so pics all blurred! But I turned this bug around to capture image when person *not* moving so steady image. This was all done in an 8 bit PIC microcontroller, I did the hardware and software for this and James Srinivasen did the memory card software and other clever bits.
Trevor Taylor did a lot of the early pcbs and hardware. I designed and built the hardware and software. There were many late solitary nights with soldering iron and oscilloscope. Stuffing all my rat's nest of hardware into a case the size of a match box was hard! It was very embarrassing when early SenseCams burst into flames due to lithium battery recharging issues and the whole of Microsoft Research (then in offices in Petty Cury, Cambridge) was evacuated.

Here is first pic taken with sensecam,

Kings College, Cambridge


Grass is pink as I took out the Infra red lens, so the chlorophyl in grass looks red.


I had about two weeks to get a demo working for a Microsoft trade show, Techfest in Seattle and Bill Gates wanted to see it! I worked late weekends, luckily my husband looked after daughter. My managers did not want to show it at Seattle, so I had to be persuasive. Bill liked it, smiled a lot, and said he would like one for his pet dog to check how it was treated when it visited the neighbours. I hand built about 18 sensecams, then my boss said could I make another 100 including some for training American soldiers in Iraq as mentioned here? I said there are better people than me to do production engineering, suggested a company and that I didn't support the war in Iraq. (My boss was North American, I am British) At my termination meeting in 2007, this was mentioned as a reason for making me redundant. I left and I did next version of sensing camera, which I am testing today. Loopholes in my patent allow new inventions.
I must mention the time I was honoured by Gordon Bell visiting my office around 2000. Gordon said, never throw any emails away, this was a flashbulb memory moment for me, I was sat in my chair and Gordon stood at my office door. Gordon has continued to be of great support to me and my memory projects before and after I left Microsoft, thank you. Some of Gordon's and Jim Gemmell's work with SenseCam covered in this book Total Recall.
Also was very pleased that Narinder Kapur of Addenbrookes approached me to help with memory problems (wanted SmartQuill but I suggested SenseCam) and Emma Berry who did the neuro trials with patients with only a few hours of memory.
Microsoft Seattle helped me with later case designs.

Today, (16th Oct) I am happily testing my Sensebulb, a sensing lightbulb that sends text messages if people have kitchen accidents, fall over, leave the cooker on, really for Alzheimer's . Chris Curry, who set up Acorn Computers after parting with Clive Sinclair, approached me re SenseBulb, and I am happy to be behind an oscilloscope debugging my software.

Here are some of the my SenseCam videos click here

I will update this blog in case I have forgotten anyone, email me at sensecam@gmail.com .
Thanks also to Jack Schoefield of The Guardian for this article about my work.

Anyway, me and my team of trusty hardware/software engineers, many from Microsoft, past and present, are available at Girton Labs if you want help or questions with your latest projects. I am at sensecam@gmail.com, or cell +44 (0) 7970 101578.




* SmartQuill telephone patent with sensors? What happened to this invention of mine from 1997? Apple bought the patent in 2008 as a lot of the technology of this patent in the iPhone.

Lyndsay Williams

www.girtonlabs.com





Thursday 15 October 2009

Microsoft to license Sensecam

I am very pleased indeed to see the SenseCam become a product, this is great news for customers. I invented the first Sensecam in 1999, wanting to help people with severe memory problems. It is very good news that Microsoft Research can license this technology.
Licensed to OMG, Oxford Metrics Group, who do the special effects for the Harry Potter films.
OMG shares went up 26% on announcement of this deal with Microsoft:
OMG and Microsoft have been working on this deal since June 2009.
Official Announcement here
OMG contacted me in April 2008 asking if I was interested in working with them, I did wonder why!


15 October 2009
OMG plc
(“OMG” or the “Group”)
IP License Agreement with Microsoft
- Vicon signs license agreement with Microsoft to develop new medical technology -
OMG plc, Oxford Metrics Group (LSE: OMG), (“OMG” or “the Group”) the technology group providing
image understanding products for the entertainment, defence, life science and engineering industries
announced today that it has signed an intellectual property (“IP”) license agreement with Microsoft
Corp.
The agreement will allow the Group’s Vicon arm, which develops motion capture products for the life
science, engineering and entertainment industries, to manufacture and sell devices incorporating
Microsoft’s SenseCam technology worldwide.
SenseCam is a wearable digital camera that takes photographs automatically, without user
intervention. Developed by Microsoft Research Cambridge, the device is most commonly used by
medical researchers as an aid for people with memory loss, but has great potential in a number of
different application areas. Research into these areas will be possible now that it will be more widely
available.
Vicon will unveil a new device, which is based on the existing SenseCam technology on its stand
#1413 at the annual Neuroscience Conference in Chicago on 17-21 October.
Ahead of the device launch OMG plc Chief Executive Officer, Nick Bolton, said:
“I’d like to thank Microsoft and Microsoft Research Cambridge for allowing Vicon to get involved in
such an important product that can support a variety of researchers, including those investigating
memory-loss. It’s an honour to work with Microsoft and to build on the amazing work they have done
with SenseCam over the past five years.
“This IP license agreement means we’re the first company to be able to deliver Microsoft’s ground
breaking technology to a wider audience. All of us at OMG look forward to working closely with
Microsoft and the researchers that have been using SenseCam Project technology and continuing to
move it forward.”
Andrew Herbert, Distinguished Engineer and Managing Director, Microsoft Research
Cambridge said:
“Microsoft’s willingness to license intellectual property arising from its research enables partner
companies to bring innovative products to market. This agreement represents the next stage in the life of SenseCam and will bring the device to a broader community.”