This article is from WeChat official account: chi things (ID: zhidxcom) , author: letter Instrument, head picture from: Vision China
According to foreign media reports, Apple recently disclosed an interesting new patent. Apple is studying how to manipulate virtual controls in the real world. In the future, the wearer of Apple AR glasses can display any surface as Touch screen, used to perform some basic operations.
The wearer of Apple AR glasses can display any surface as a touch screen
I have to say that this operation reminds me of the floating electronic screen that appears in science fiction movies by the big boss at any time and anywhere. In fact, Apple’s patent was filed as early as 2016, but it was only recently exposed. Imagine if this technology is realized in the future, can we just wear a pair of glasses with us, so that tablets and mobile phones do not need to be hung on our bodies all the time?
The virtual screen in the movie “Avengers”
The patent output of Apple, a “big patent owner”, can be described as a steady stream. Last month’s Apple Developer Conference (WWDC20) demonstrated the patents of “iPhone on BMW” and “Spatial Audio Surround Sound”. Eye-opening, and Apple’s interesting patents are far more than that.
The brain holes of Apple’s patents are always unexpected, and these patents are constantly enriching Apple’s major products, which are involved in iPhone, Mac and even the upcoming AR/VR headsets.
For the products we use in our daily lives, what patents has Apple recently launched that have a brainstorming but not inferior practicality? Once these patents land, what kind of wonderful experience will they add to us?
We have screened out 14 interesting patents that Apple recently exposed. Some of these patents have been applied to Apple’s products, and some have not yet landed, but they may change the shape of Apple’s future products. So what are the incredible features of these patents? Let’s look at them one by one.
Eye tracking, pupil expansion, Apple’s investment in VR/AR is not limited to glasses
1. Realistic surface can be changed into touch screen
At the beginning of the article, it was mentioned that the newly exposed Apple patent is related to the “Apple Glass” being developed by Apple. The name of the patent submission is “Method and device for detecting touch between a first object and a second object< span class="text-remarks" label="note">(Method and device for detecting a touch between a first object and a second object)“, on the actual landing, seeIt seems that Apple wants to use it as a virtual control superimposed on real objects.
In this patent, Apple solves a problem related to “how people wearing AR glasses interact with the virtual environment they see”.
The patent shows that the natural way for humans to interact with objects is to touch with their hands, but the displays of AR glasses are getting smaller and smaller, and the lenses are getting closer to the user’s retina, which makes the behavior of AR glasses players touching the screen change. It is extremely difficult and inconvenient.
If the vision of this patent is realized, users can touch the virtual screen they need on any plane. For example, if the AR glasses confirm that the user has touched a real surface that the user considers to be a button, the AR glasses system can react on its own as if the user actually clicked the button control.
The patent shows how to make a flat object into a calculator
This idea sounds difficult. Apple has tried to directly allow users to interact with the AR environment before, but such operations require hardware such as gloves or finger sensors. At the same time, trying to visually monitor the contact of the finger with the virtual surface is not accurate enough.
Apple’s idea is to detect when users touch real-world objects through infrared heat sensing. This method allows Apple Glass to visually project controls onto real-world objects, and when the user touches these objects , React by sensing the heat conduction when touching an object.
Different display images through different temperatures
Although we still cannot confirm whether this patented technology will be integrated into Apple’s future AR glasses products, this insight is enough to make us imagine that the phenomena in the science fiction world may one day be realized in the real world.
2. Infrared light source eye tracking makes VR shooting games aim more accurately
In the newly released list of Apple patents granted by the US Trademark Office on June 30 this year, there is a patent that introduces the eye tracking invention that Apple plans to carry in future headsets. This invention is equipped with a blink assisted gaze tracking function to provide higher visual accuracy and clarity.
Users will feel immersive in the VR(virtual reality) experience In the past, the VR system can show the user a three-dimensional scene to generate the feeling of being in it. When the user views the image through the VR system, he may feel that he is experiencing the movement and change of the surrounding environment from a first-person perspective.
Eye tracking system front view example
Side view example of eye tracking system
Similarly, MR(mixed reality) or AR(augmented reality ) The system also combines computer-generated virtual content with real world views to add visual content to the real world where the user is.
Therefore, such headsets must provide accurate data on the user’s view to ensure that the user sees the correct image. If the image that the user sees in the shooting game is not accurate, then there is no way to accurately aim and shoot, and this “pot” user can’t remember it himself.
This patent from Apple proposes a scheme that uses the blinking of an infrared light source to track the eye. Specifically, this eye tracking solution is equipped with a camera module at each of the left and right eye positions of the head-display user to emit a light source to the eyeball. The eye tracking camera can be placed on the opposite or on the same side of the eyeball according to the specific situation, and finally the light reflected by the eyeball is mapped to the camera.
It is understood that the refresh rate of this eye tracking camera is between 60fps and 120fps. After the eye tracking information is generated from the captured eye data, the control unit will predict the gaze of the eye based on the optical detection data and calibration data. 3D data of points, corneal center, etc., and then re-correspond to the visual axis of the eyeball.
For human eyes, pupil center, corneal centerMathematical model of location estimation
3. Miniature holographic optical projection changes AR field of view and image quality
In a patent application related to future AR glasses released by Apple on June 18 this year, a “pupil expansion function” was mentioned, which aims to enlarge The field of view of the image projected onto the optical lens to provide higher quality images for AR applications.
Pupillary expansion function icon
This patent called “Pupil Expansion” mainly relates to the expanded light beam in the optical reaction device. The patent document shows that the holographic optical element can be used in the waveguide device of the head-mounted display device.
In the past, the way to expand the field of view of AR images mainly included the use of conventional optical devices such as reflection, refraction or diffracted light, but the integrated design using these methods tends to be bulky and has certain inherent limitations. For example, a conventional mirror may need to have a reflection axis that must be coincident with the discovery, so that the mirror orientation is poor, or the reflection output at the mirror is blocked.
Apple’s pupil expansion technology involves one or more improved methods for beams from the incident field of view to the exit field of view. This method can be used in optical equipment to expand the vision of the projected image. field.
Optical system diagram
4. Stickers that can feel the touch texture of virtual images
Apple’s patent investment for AR glasses is not only in the glasses itself, but also in patents on accessories. On June 4 this year, the United States Patent and Trademark Office issued a patent for a “finger sensor device” filed by Apple to match future AR glasses and other head-mounted display devices. Prior to this, Apple has applied for 5 finger device patents to match the use of future head-mounted display devices.
The finger device applied this time is similar to a ring sticker, equipped with sensors that can feel force, images, etc., so when users interact with real-world objects, they can feel the real world through one or more sensors Physical information of the object.
A top view of the finger device
Through this sensor, visual, audio, tactile and other content can be presented on the computer and output through tactile. Therefore, through the head-mounted display device, users can feel the surface contour, texture, object color and other visual attributes of real objects, as well as other physical attributes such as temperature, sound, force, and weight.
In addition to the headset or AR glasses, this finger device can also be used with MacBook, iMac, iPad, iPhone, Apple Watch, etc.
Finger device cross-sectional side view
The iPhone replaces the passport car key. Is it feasible to only take a mobile phone to go out in the future?
Apple has always had the ambition to “make the iPhone the only device that people need to carry when going out”. Apple has replaced cash with Apple Pay, replaced U disk with AirDrop, and the next step is to use iPhone to replace passports. And car keys.
1. Replace the passport with an iPhone, and the status of the physical passport is not guaranteed?
Currently, mobile phones have allowed us to go out without having to carry papers and pens, calculators, cameras, mirrors, and even wallets. Next, the tech giants have turned their attention to “electronic identities” so that people don’t have to remember to ask for them. Bring your documents.
As early as March 2017, Apple filed a patent application with the United States Patent and Trademark Office entitled “Incorporating Documents into a Secure Environment”. The content was that Apple wanted the iPhone to replace the ID card.
This patent describes the use of iPhone such as NFC(Near Field Communication) or RFID(Radio Frequency Identification Technology)’s short-range radio function to store personnel identity credentials embedded in integrated circuits.
In this way, iPhone can obtainCredential information, such as ID card, driver’s license, passport, etc. The iPhone can store the user’s personal name, date of birth and other information to verify the user’s true identity.
Add identity information to iPhone
When the relevant department personnel request the corresponding certificate information, iPhone users can use passwords, fingerprints, Face ID and other methods to unlock the device, and then show the personal certificate information stored in the phone to the staff. In this way, iPhone can send the user’s personal data to the relevant department personnel of the required information to complete the identity verification.
Later, Apple also cooperated with the governments of Japan, Germany, and the United Kingdom to launch ID card projects in 2019.
iPhone to replace ID card patent operation method
In July of this year, Apple’s latest patent application entitled “Provide Verified User Identity Proof” detailed the method by which the user ID can be recorded or transmitted and confirmed.
Although Apple does not use the word “iPhone”, it can emphasize that this technology may be used in any device, but “Apple plans to replace passports, driving licenses, library cards, ski resort members and other entities with iPhones. The message of “identity proof” is widespreadBiography, even bombed by netizens on hot searches.
iPhone replaces hot search on passport
This new patent document details how to copy and use user identification. According to Apple’s vision, devices such as the iPhone can safely transmit certain identifications. If the user’s identity has been verified, the biometric sensor of the device such as Apple Watch can be used to directly send the identity certificate.
Information verification transmission path
However, whether the iPhone replaces a physical ID card or a passport, there are difficulties in the actual landing. In addition to the need to be certified by the government, we must try our best to conform to the identity verification standards of various countries. Each country has different ID standards, and most countries currently do not allow “network certificates” to replace physical certificates, so “iPhone replaces physical certificates.” The vision is long and long.
2. iPhone “car key” can drive BMW
At this year’s WWDC, the iPhone has officially become the “debut” of the digital car key. iPhone canTo unlock the car through the NFC function, you can also lock the car remotely, and you can “lend” the car key to others through the APP anywhere.
iPhone driving BMW
It is said that this technology uses Apple’s self-developed U1 chip, which can accurately detect the distance of the device and ensure safety.
The patent of “iPhone as a car key” can be traced back to 2013. In April 2013, Apple released a report entitled “Apple Reveals Advanced Car Access Control System”, and this report was originally Submitted to the United States Patent and Trademark Office in 2011, the patent shows that the future iPhone will be able to open the door, start the engine, turn off the engine at a specific time, and be able to carry out personalized programming settings for the car.
Examples of patent-related vehicles and flow chart of operating vehicles through equipment
Since then, related patent documents in 2016 show that Apple users can control cars through portable devices such as iPhone or iPad, including common operations such as unlocking the doors, starting the engine, adjusting the temperature, and opening the car’s headlights and sunroof.
In 2019, Apple has applied for a new patent for “Vehicle Authorization System and Method”. This system will provide communication between the vehicle and iPhone holders.Communication allows users to perform certain tasks, such as opening car doors.
It is reported that this system will use biometric technology to confirm the identity of the user and determine whether the user is authorized to enter and operate the vehicle. In addition, the owner can authorize others to enter and control the vehicle.
It is said that Apple may also apply this technology to self-driving car-hailing services and provide authentication related to mobile payment methods such as Apple Pay.
“Vehicle authorization system” can be understood as a system-level interface. In the future, as long as it is a device that uses Apple smart car hardware, it can be seamlessly connected to this interface. At that time, mobile phones can truly become car keys. Ontology, instead of using the Internet of Vehicles system as an intermediary to perform tedious operations.
Knob “mouse”, glass keyboard, the highlight of Mac is not only the self-developed chip
1. The knob control device beyond Surface Dial
Among the 63 newly granted patents issued by the U.S. Patent and Trademark Office on July 7, there is a future accessory that may be used with devices such as iPad, Mac, iPhone, and automobiles.
This patent describes a knob control device system. The mystery is that Apple has never published this device as a patent application before, and we can only understand this system when this patent is granted.
Apple’s patented knob control device
The knob control device can be removed from the display or other forms of surface, and then continue to be placed on other equipment to control the equipment.
When this knob device is placed on the display, the display will emit light according to the placement position of the knob, and the display can also display scale marks or other scale information on the periphery of the knob.
This is very similar to the desktop Surface Dial accessory of Microsoft’s Surface Studio. Surface Dial is placed on the display screen, and information is displayed around the knob, so users can see the information at a glance.
Microsoft Surface Dial use demo
Apple’s knob control device can be used to adjust the media playback volume, adjust the temperature of the thermostat, control the selection of tracks in the music library, and so on.
The main reason that Apple’s patent improves the existing Microsoft knob device is not to limit the use of this device to a single display or device.
This patent description shows that if necessary, the knob can even be connected to other suitable structures such as exterior wall surfaces, vehicles, computer housings, refrigerators or stove surfaces.
Apple’s patented details of the knob control device
Although the current knob control device is circular, Apple’s patent shows that it can also have other shapes, andIn the future, other functions will be added, such as sliding controls, adding buttons, and even mounting a display in the middle of the knob.
2. Glass keycaps reduce abrasion and thickness
Following the widespread complaints about the dust accumulation problem of the MacBook butterfly keyboard, Apple had to reuse the old scissor keyboard. In addition to the scissor keyboard, Apple has also developed a special “glass keycap”.
On July 2 this year, the United States Patent and Trademark Office issued Apple’s patent application for “transparent keycaps”. The new patented keycaps use a fully transparent design scheme, which will use glass, fully transparent ceramics, and sapphire glass. It is made of transparent or transparent materials. The keycaps made of this type of material have high bending rigidity and compressive strength, even if the keycaps are made thinner, there is no need to worry about bending and deformation. In addition, the surface of the “glass keycap” will be polished, so scratches and defects can be avoided.
Glass keycap patent icon
The “glass keycap” button logo will be printed on the lower end of the button, so even after repeated rubbing, the letters on the keyboard will not worry about being worn.
In addition to durability, the patent also mentions that this keycap is light-transmissive, allowing the keys to show different colors and brightness through RGB and LED. In this way, whether it is day or night, the light of the “glass keycap” can adapt to the user’s eyes and provide a comfortable experience.
The back touch patent was only obtained in 15 years, and the writing touch can be passed through the Pencil
1. The back touch patent is a one-handed tablet computer
On June 23 this year, Apple finally obtained a patent for the back touch of a tablet computer. This touch patent named “Handheld Electronic Device with Multiple Touch Sensing Devices” was filed in 2005 and has passed In 15 years, you must know that the iPad was not available at this time. It seems that this patent was not necessarily designed for the iPad at that time.
The idea is to allow users to hold the tablet while still using this hand to perform basic operations such as “close or switch web pages” on the back of the tablet without having to vacate another Hands perform task operations on the tablet’s display.
Patent touch icon on the back
This proper design is the gospel of “lazy cancer”. No wonder some people say that “lazy people promote technological progress”. This “lazy artifact” patent is a good proof.
As Apple Acrade becomes more and more popular, the iPad with back gaming controls may also have a place in the tablet market in the future. What’s interesting is that after 15 years, this project is still effective and still developing, and we still don’t know whether this project will land.
2. Feel the texture of painting with Apple Pencil
In December last year, Apple issued a patent application report titled “Writing on the iPad is like writing on real paper, and using a brush tool is like drawing on a real canvas.” This patent application was approved on June 23 this year.
This patent relates to the advanced tactile technology that Apple Pencil may use in the future, which will provide users with the feeling of writing or drawing on textures. This is not the first time Apple Pencil has stepped into the patent for providing texture and tactile feedback. This project started as early as 2015.Two similar patents have been published.
From the picture, we can see that the patent provided in 2015 contains a sensing device called “rumble pack”. The stylus can generate tactile feedback through contact with a tactile feedback device in the form of a “wired” on the iPad.
The first generation tactile feedback device equipped with “rumble pack” sensor device
When the stylus slides over the screen of the tablet, the stylus will sense the image, and the corresponding texture information will be transmitted to the “rumble pack” through the software on the tablet, so the user can observe visually The image on the tablet, and you can also feel the texture of the image.
In the latest patent, the “rumble pack” is integrated into the Apple Pencil. The stylus can feel tactile feedback through the tip of the pen to simulate drawing on textured paper with a stylus. Integrating a tactile feedback device into the pen tip can more accurately simulate the feeling of writing on a textured surface.
Improved patent illustration of tactile feedback device equipped with Apple Pencil
In addition to simulating the feeling of writing on paper, this system can also simulate the feeling of writing and painting on different surfaces such as paper and canvas based on different programming and with different contours selected by the user.
In addition, this device can also simulate different types of writing and painting tools such as pencils, ballpoint pens, fountain pens, highlighters, brushes, etc. based on programming and user-selected configuration files.
Apple Watch becomes a “gamepad” and can calculate riding wind resistance< /span>
1. Patented vein scanning technology enables players to play games “bare-handed”
In April 2019, Apple applied for a new version of Apple Watch called “Apple invented a new version of Apple Watch that can unlock more gesture functions in VR games”. This patent was approved on June 30 this year.
This patent shows that the future Apple Watch can be equipped with a sensor or a micro camera at the bottom of the strap to capture the user’s vein image, and then automatically recognize the user’s specific gestures in the VR/MR game.
Then Apple Watch will input the gestures and finger position commands recognized by vein movement into the system and perform actions. In this way, users no longer need to use voice or touch input, but can enter commands for Apple Watch only through normal vein movement.
Patent perspective view of Apple Watch strap with multiple sensors
Apple’s efforts in vein recognition go beyond this. In the 50 patents granted to Apple issued by the United States Patent and Trademark Office yesterday, it covers a “Vein Matching for Difficult Biometric Authentication Cases”. “The patent.
Apple recognizes facial vein patent
This patent proposes that veins can be used as a recognizer. Although facial features are easy to replicate, the pattern of veins varies greatly from individual to individual, even for twins. Therefore, it is extremely difficult to create a fake face with consistent vein structure.
This facial recognition system will use sub-skin image technology to create a 3D image of the user’s veins, just like the infrared sensor in the camera, which can capture flood and spot patterns from the infrared illuminator that illuminates the user’s face.
If this face recognition method is implemented, it can greatly reduce the error rate of facial recognition.
2. Good news for cycling enthusiasts: Apple Watch’s new patent can calculate cycling wind resistance
In March 2017, Apple issued a patent titled “Apple Watch invented a system designed to display cyclist activity and monitoring data.” This patent was approved on June 30 this year, focusing on Calculate information such as wind resistance and heart rate of the rider during cycling activities.
Cyclists are generally interested in the power output and calories burned by their cycling activities. When the rider is riding, three main factors lead to energy consumption, including frictional resistance between the bicycle tire and the ground, gravity, and air resistance.
Apple’s new patent analyzes the resistance experienced by the rider during the cycling movement to determine the rider’s total consumption.
This patent will carry a variety of microprocessors, displays and sensors on the Apple Watch to analyze and measure specific conditions such as heart rate, position, and direction.
At the same time, the accelerometer in the device will also track the acceleration changes during riding, and the device can automatically adjust the data according to the different positions where the rider wears the Apple Watch.
Finally, this new patented device can combine measurements such as heart rate, speed, altitude, and riding position to estimate the rider’s total power output and the proportion of each part of the output power.
In some different device models, the motion sensor can also be equipped with a speedometer, gyroscope, barometer, magnetometer or compass, etc., and can also optimize the lower power consumption parts through the motion coordination processor to make The device achieves the effect of continuously processing motion sensing.
The force analysis of cyclists and Apple’s new patent to determine the wind resistance estimation method
Spatial audio patent can enjoy stereo surround sound by yourself
This year’s WWDC20 demonstrated a “Spatial audio spatial audio” patent that can be used for AirPods Pro and Apple TV. The purpose is to allow users to use headphones to feel immersive through the surround sound system. The auditory experience.
“Spatial audio spatial audio” patent
Prior to this, 3D surround sound effects usually appeared in AR, VR, and MR, and Apple’s new patent also gave the surround sound effects to headset users.
Because human heads and mobile phones connected to headphones cannot remain still at all times, Apple has created a virtual sphere to replicate the 3D sound experience of the theater.
By adding an audio filter to the headset, fine-tuning the frequency received by each ear, and associating multiple channels of the audio object with their respective distribution spaces, and performing mixing operations, users can Feel the sound coming from any position.
In addition, this system also uses the acceleration sensor and gyroscope on AirPods Pro to track the movement of the head and remap the sound field, so that the sound can be firmly fitted to the device, and the human ear is the center of the production. Realistic “cinema-level experience”, this function supports 5.1-channel and 7.1-channel multimedia files, and also supports “member exclusive” Dolby Atmos.
Space Audio Patent Icon
Conclusion: Apple’s patent ecology is closely linked
According to the statistics of the U.S. Patent and Trademark Office (USPTO) and the patent company IFI, the technology giant Apple, as a “big patent owner”, has recently In the past few years, more than 2,000 patents have been obtained every year, but this number is only ranked 10th in the number of patents in the United States.Around the time, a steady stream of patent applications can’t even make up the limited number of granted patents.
Apple’s patents can also be described as a chain: the patented knob control device is matched with various displays, the Apple Pencil increases the writing touch of the iPad, and the finger sensor prepares for the AR glasses… The layout of Apple’s ecology can be described as comprehensive.
The dozen or so patents mentioned in the article are just the tip of the iceberg where Apple has applied for a large number of patents. Patents like “spatial audio” have already been implemented on products, bringing changes and improvements to existing products, and similar to “back Patent such as “touch” may not land, but it can also be seen that Apple, a technology giant, has been innovating consumer products in a certain period of time.
Apple patents keep breaking news, and we continue to look forward to what kind of wonderful experience these patents will eventually appear in different products.
Source: PatentlyApple, AppleInsider
This article is from WeChat official account:Smart things (ID: zhidxcom), author: instrument channel