text:
US9172747B2 - System and methods for virtual assistant networks - Google PatentsSystem and methods for virtual assistant networks Download PDF
- Publication number
- Authority
- United States
- Prior art keywords
- virtual assistant
- dialog
- user
- request
- example
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
- 230000003993 interaction Effects 0.000 claims description 107
- 230000004044 response Effects 0.000 claims description 87
- 230000015654 memory Effects 0.000 claims description 29
- 238000000034 methods Methods 0.000 description 125
- 238000004458 analytical methods Methods 0.000 description 17
- 239000000203 mixtures Substances 0.000 description 16
- 239000002609 media Substances 0.000 description 15
- 230000006399 behavior Effects 0.000 description 8
- 230000000694 effects Effects 0.000 description 6
- 238000006243 chemical reaction Methods 0.000 description 4
- 230000001276 controlling effects Effects 0.000 description 3
- 239000003795 chemical substance by application Substances 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 230000000977 initiatory Effects 0.000 description 2
- 230000002085 persistent Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network-specific arrangements or communication protocols supporting networked applications
- H04L67/10—Network-specific arrangements or communication protocols supporting networked applications in which an application is distributed across nodes in the network
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/16—Sound input; Sound output
- G06F3/167—Audio in a user interface, e.g. using voice commands for navigating, audio feedback
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation, e.g. computer aided management of electronic mail or groupware; Time management, e.g. calendars, reminders, meetings or time accounting
- G06Q30/00—Commerce, e.g. shopping or e-commerce
- G06Q30/02—Marketing, e.g. market research and analysis, surveying, promotions, advertising, buyer profiling, customer management or rewards; Price estimation or determination
- G06N5/00—Computer systems using knowledge-based models
The instant application is a continuation of U.S. patent application Ser. No. 14/184,697, titled "SYSTEM AND METHODS FOR VIRTUAL ASSISTANT NETWORKS", filed on Feb. 19, 2014, which claims priority to U.S. provisional patent application Ser. No. 61/769,191, titled "SYSTEM AND METHODS FOR A NETWORK OF VIRTUAL ASSISTANTS", filed on Feb. 25, 2013, the entire specification of which is hereby incorporated by reference.
1. Field of the Invention
The invention relates to the field of artificial intelligence, and more particularly to the field of natural language interaction-based applications, such as virtual assistants.
2. Discussion of the State of the Art
Virtual assistants are an example of a class of software applications, referred to herein as natural language interaction applications that use natural language to interact with a human user. With respect to virtual assistants in particular, these applications perform functions similar to those provided by human assistants, in that they can engage in conversations with their users in order to for example provide information, carry out routine tasks, or perform other operations as required. Many virtual assistants have been used in association with web sites, with these virtual assistants often featuring the use of graphical images or avatars to simulate the experience of interacting with a human being.
While some natural language interaction applications known in the art may possess rudimentary natural language understanding capabilities, in reality most such applications, and particularly most virtual assistants known in the art, are tightly scripted applications that carry out a limited range of activities. While scripted agents have proven useful in many online scenarios, their acceptance has generally been limited because, when users are restricted in their ways of expressing themselves in natural language, the communication language is not natural. Any notion of the virtual assistant's behaving like a real assistant disappears because too much effort is demanded of the user when the user is expected to know precisely what to say or what to do in order to achieve her goals. Thus there is a distinct need in the art for interpreting fluid and freely expressed natural language in various applications, for example by virtual assistants. This challenge is exacerbated by the fact that, even when equipped with rudimentary natural language interpretation, natural language interaction applications such as virtual assistants that seem to only "know" a small number of facts and who only "talk about" a very limited range of subjects (such as appointment management) do not seem very realistic to humans, who are used to being able to say things in a fluid way, shifting subjects easily and intermixing casual banter with task-oriented speech.
The use of natural language interaction applications generally, and virtual assistants more particularly, has also been hampered by the high degree of complexity and cost associated with developing and deploying virtual assistants, particularly those that have any significant degree of natural language interpretation and interaction capability. As such many of the services used by virtual assistants are housed in the cloud and are connected to using a network such as the internet, a corporate intranet, Bluetooth and other networks. Maintaining a connection to these services is necessary to completing many tasks of virtual assistants. However, given that in the current environment, there are many areas where connectivity is not available or expensive due to, for example, roaming charges or networks requiring a subscription, connectivity to other services may not always be available. Moreover, since typical users have a plurality of devices, maintaining consistency across devices is essential.
Accordingly, what is needed in the art is a system and various methods that enable virtual assistants to operate in a networked fashion with one another, facilitating interaction between virtual assistants as well as the management of a virtual assistant network for efficient task handling.
Accordingly, the inventor has conceived and reduced to practice, in a preferred embodiment of the invention, various systems and methods for the implementation and operation of virtual assistant networks, including the use of a virtual assistant broker that may be used to manage a number of virtual assistants in a networked fashion.
According to a preferred embodiment of the invention, a system for operating a virtual assistant network (VAN), comprising a plurality of virtual assistants (VAs) such as, for example, SIRI™ or other virtual personal assistants, or other assistive products or services, a plurality of VA brokers (that may be stored and operating on network-connected electronic devices), and a VA communication protocol (such as a standardized markup language or other means of standardizing communication between virtual assistants and brokers), is disclosed. According to the embodiment, VAs may be connected to and in communication with one another (as well as with brokers and human users) via the Internet or another communication network, such as a cellular data network or any other such appropriate means for electronic communication. A VA broker may be in communication with a plurality of VAs for such purposes as to receive requests or information as well as to send requests or other information to VAs, thereby facilitating a VAN by unifying communication between individual virtual assistants in an orderly fashion. For example, a virtual assistant may be assisting a user that is looking for a restaurant in a foreign country, and may send a request to a VA broker for information on local restaurants. The VA broker may then identify another virtual assistant in the network that may be able to process this request (for example, a virtual assistant maintained by a company in the country that is designed to assist local individuals in locating restaurants and placing reservations), and submit the request to that assistant on the user's behalf. The VA broker may then receive the result of the query (such as a list of nearby restaurants and their associated information), and send this back to the originating virtual assistant for presentation to the user. Alternatively, in some embodiments the broker assembles a list of virtual assistants that are likely to be able to satisfy a user's request that was submitted by the user's host virtual assistant, and send this list of possible virtual assistants back to the host virtual assistant. The host virtual assistant may then select one of the virtual assistants from the list provided by the broker and contact that virtual assistant with the request directly (typically, using contact information such as IP address that is provided by the broker with the list of eligible virtual assistants). In other embodiments, the host may exercise more direct control, selecting a virtual assistant and connecting the host virtual assistant to the selected virtual assistant, or even receiving responses to the user's request from a plurality of virtual assistants and returning either a list of responses (with or without information concerning which virtual assistant provided each response) or a single response selected by the broker. In this manner, a network of virtual assistants may be utilized to expand the functionality of any one particular assistant without any additional complication on the part of the user, who may perceive that they are consistently interacting with a single virtual assistant with which they may be familiar, while networked operation takes place "in the background" without their knowledge.
Further according to the embodiment, a VA communication protocol (such as, as envisioned by the inventor, a standardized markup language) may be utilized to facilitate interaction between virtual assistants and brokers. Ordinarily, each particular virtual assistant or broker may have a particular means of interaction, that may not be interchangeable with another. Such behavior is common practice in various corporate technologies in the art, where proprietary technologies may impede interaction between products or services from different vendors. By utilizing a standardized protocol, interaction between these disparate technologies may be facilitated in a structured and reliable manner, such that networking functionality and the use of brokers (that may need to interact with a large variety of different technologies to perform their functions) become possible according to the invention.
According to the embodiment, not all inter-VA communication may need to utilize a broker. For example, it is possible that a particular VA may already be familiar with another VA that is appropriate for a particular request. In such a scenario, a user may interact with their familiar VA that may serve as the "host" for the interaction. As needed, the host VA may communicate with a known "target" VA for assistance with specific requests. The target VA may then fulfill those requests as they would during normal operations (as if the requests had been received directly from a user), and return the results to the host VA for presentation to the user. In this manner, networked functionality may be provided by virtual assistants when no broker is available, or when it may be undesirable to utilize one (such as when the host VA already knows of a specific target VA that may be appropriate for the request).
Further according to the embodiment, a VA broker may communicate with another VA broker as necessary, such as to utilize virtual assistants outside of its own network. For example, a virtual assistant network (VAN) may be maintained for a specific region or country, encompassing a variety of virtual assistant technologies relevant to that specific locale. When a user submits a request for information from another region, such as asking for local news from another country, it may be possible that no particular VA within the network is ideal for handling the request. In this scenario, the originating "host" VA may decide to communicate with a VA broker for assistance in fulfilling the user's request. The VA broker may then reach out to another "target" broker for the specified region to pass the request to their network for handling, and in turn the target VA broker may submit the request to a local virtual assistant (as described previously) for handling. The results of that interaction may then be returned to the host VA for presenting to the user. In this use case, the user may still perceive that they are only interacting with their familiar virtual assistant, while all brokering operations take place without their knowledge.
In another preferred embodiment of invention, a method for providing networked virtual assistant operations is disclosed. According to the embodiment, a user may submit a request to a virtual assistant for fulfillment. According to the nature of the specific request, the VA may identify that another VA is needed to fulfill the request (or that utilizing another VA for assistance may further enhance the fulfillment of the request). The VA may then send a request to a VA broker for assistance, the broker then communicating with another target VA for fulfillment of the request. The results of the request may then be returned to the host VA for presentation to the user. Alternatively, in some embodiments the broker assembles a list of virtual assistants that are likely to be able to satisfy a user's request that was submitted by the user's host virtual assistant, and send this list of possible virtual assistants back to the host virtual assistant. The host virtual assistant may then select one of the virtual assistants from the list provided by the broker and contact that virtual assistant with the request directly (typically, using contact information such as IP address that is provided by the broker with the list of eligible virtual assistants). In other embodiments, the host may exercise more direct control, selecting a virtual assistant and connecting the host virtual assistant to the selected virtual assistant, or even receiving responses to the user's request from a plurality of virtual assistants and returning either a list of responses (with or without information concerning which virtual assistant provided each response) or a single response selected by the broker.
The accompanying drawings illustrate several embodiments of the invention, and together with the description, serve to explain the principles of the invention according to the embodiments. One skilled in the art will recognize that the particular embodiments illustrated in the drawings are merely exemplary, and are not intended to limit the scope of the present invention.
The inventor has conceived, and reduced to practice, various systems and methods for operating virtual assistant networks.
One or more different inventions may be described in the present application. Further, for one or more of the inventions described herein, numerous alternative embodiments may be described; it should be understood that these are presented for illustrative purposes only. The described embodiments are not intended to be limiting in any sense. One or more of the inventions may be widely applicable to numerous embodiments, as is readily apparent from the disclosure. In general, embodiments are described in sufficient detail to enable those skilled in the art to practice one or more of the inventions, and it is to be understood that other embodiments may be utilized and that structural, logical, software, electrical and other changes may be made without departing from the scope of the particular inventions. Accordingly, those skilled in the art will recognize that one or more of the inventions may be practiced with various modifications and alterations. Particular features of one or more of the inventions may be described with reference to one or more particular embodiments or figures that form a part of the present disclosure, and in which are shown, by way of illustration, specific embodiments of one or more of the inventions. It should be understood, however, that such features are not limited to usage in the one or more particular embodiments or figures with reference to which they are described. The present disclosure is neither a literal description of all embodiments of one or more of the inventions nor a listing of features of one or more of the inventions that must be present in all embodiments.
Headings of sections provided in this patent application and the title of this patent application are for convenience only, and are not to be taken as limiting the disclosure in any way.
Devices that are in communication with each other need not be in continuous communication with each other, unless expressly specified otherwise. In addition, devices that are in communication with each other may communicate directly or indirectly through one or more intermediaries, logical or physical.
A description of an embodiment with several components in communication with each other does not imply that all such components are required. To the contrary, a variety of optional components may be described to illustrate a wide variety of possible embodiments of one or more of the inventions and in order to more fully illustrate one or more aspects of the inventions. Similarly, although process steps, method steps, algorithms or the like may be described in a sequential order, such processes, methods and algorithms may generally be configured to work in alternate orders, unless specifically stated to the contrary. In other words, any sequence or order of steps that may be described in this patent application does not, in and of itself, indicate a requirement that the steps be performed in that order. The steps of described processes may be performed in any order practical. Further, some steps may be performed simultaneously despite being described or implied as occurring non-simultaneously (e.g., because one step is described after the other step). Moreover, the illustration of a process by its depiction in a drawing does not imply that the illustrated process is exclusive of other variations and modifications thereto, does not imply that the illustrated process or any of its steps are necessary to one or more of the invention(s), and does not imply that the illustrated process is preferred. Also, steps are generally described once per embodiment, but this does not mean they must occur once, or that they may only occur once each time a process, method, or algorithm is carried out or executed. Some steps may be omitted in some embodiments or some occurrences, or some steps may be executed more than once in a given embodiment or occurrence.
When a single device or article is described, it will be readily apparent that more than one device or article may be used in place of a single device or article. Similarly, where more than one device or article is described, it will be readily apparent that a single device or article may be used in place of the more than one device or article.
The functionality or the features of a device may be alternatively embodied by one or more other devices that are not explicitly described as having such functionality or features. Thus, other embodiments of one or more of the inventions need not include the device itself.
Techniques and mechanisms described or referenced herein will sometimes be described in singular form for clarity. However, it should be noted that particular embodiments include multiple iterations of a technique or multiple instantiations of a mechanism unless noted otherwise. Process descriptions or blocks in figures should be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps in the process. Alternate implementations are included within the scope of embodiments of the present invention in which, for example, functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those having ordinary skill in the art.
As used herein, "virtual assistant" means a software, hardware, or hybrid hardware/software application that provides assistance to humans analogous to that provided by human assistants, generally using an intuitive user interface, such as one that allows a user to express her needs in natural language.
As used herein, "natural language interpretation" means interpretation, by a machine, of one or a series of utterances made by a human user speaking or writing naturally; in some cases "natural language" may be supplemented by gestural, haptic, or other input combined with natural language input (for example, saying "move this here" while pointing or clicking on an object and a later a location). That is, a "natural language interpreter" generally does not require users to format their requests in simple, fixed patterns (such as a program might be required to do, as for example when invoking a web service or an application programming interface (API)).
As used herein, a "dialog" is a conversational exchange between two or more participants consisting of a generally unstructured sequence of natural language contributions in the form of requests, responses, statements and the like. A dialog is an interaction sequence that actually occurs (i.e., in run-time) during a conversation exchange between a user and a natural language system.
As used herein, a "flow" is a logical task unit comprising a set of related interaction steps required (and preconfigured) in order to perform a task. Typically a flow may comprise one or more transition conditions and operative nodes. A flow is typically defined in a natural language system as a series of predefined steps to perform when some triggering condition is satisfied.
As used herein, an "instructional path" is a sequence of instructions from one or more flows that were invoked or executed during the delivery of a dialog by a natural language system. In effect, an instructional path is the realization of one or more flows that occurs in an actual conversation (or dialog) between a user and a natural language system; it represents the sequence of decisions (or instructions) taken in support of natural language conversation (whereas the corresponding dialog is the actual sequence of what was "said" during the conversation by the parties involved, rather than why it was said).
As used herein, a "natural language interaction system" or a "natural language system" is a computer-based system capable of interacting via dialogs with one or more users using natural language (perhaps augmented by haptics, gestures, or other nonverbal means). Natural language systems carry out their function using dialogs, which can be thought of also as the actual, as executed, sequence of interaction steps between a user and a natural language system. A natural language system may participate in a dialog, by following the instructional steps of flows.
As used herein, a "language object" means an abstract representation of a logical unit of human linguistic expression that has meaning and is suitable for processing by automated systems such as virtual assistants. Language objects, in their simplest form, are represented as single words that represent a plurality of variants of a single common meaning, including inflectional variants and variants connected by synonymy. That is, generally a language object represents all variants and synonyms of the core word that represents it, and language objects may contain grammatical variants of words as well (such as verb tenses, contractions, and so forth).
As used herein, "script" means an executable sequence of instructions, generally stored (or at least rendered for execution) as plain text, which can be executed by an appropriate script interpreter. Scripts may be written in any scripting language known in the art, such as Perl, Python, PHP, Ruby, Groovy, and the like, or in any similar scripting language including proprietary scripting languages. As described variously herein, scripts generally are executed upon occurrence of specific events, such as initiation or termination of a user interaction session, or the triggering of a business rule such as "execute the following script if the number of active sessions exceed a configured number". Scripts can be nodes in flows (in which case they are referred to as "script nodes").
As used herein, "request" or "question" means a user-supplied or automatically supplied input to a natural language interaction-based system such as a virtual assistant, or to a virtual assistant broker.
As used herein, "response" means a system-supplied output from a natural language interaction system (as opposed to a user-supplied or automatically supplied input).
As used herein, "condition" means a rule or logical expression used for interpreting natural language utterances that, if satisfied, will allow for execution of one or more actions or will generate one or more effects. There are several types of conditions envisioned according to the invention, comprising for example: language recognition rules, skip conditions and transition conditions.
As used herein, a "language recognition rule" is a specific type of condition built up from language objects and used for capturing natural language expressions. For example, a language recognition rule can be used to interpret or capture the intention of a user request.
As used herein, "trigger" or a "flow trigger" is a construct which holds a condition, and which makes the system activate and start executing a flow if the condition is fulfilled.
As used herein, "listener" means is a rule construct with a conditional part and an operational part. When the conditional part is fulfilled, the operational part is invoked or executed, whereupon it updates current state based on information present in the current state and in the natural language input (the user request). Listeners may be configured to operate on a global level, on a flow level, or on a transition level, and they are generally evaluated in that order.
As used herein, "transition" or "transition condition" means a condition that determines whether the system should pass a transition to enter a new instructive step or process, a node (or a new transition). Transition may be input consuming (requires user input) or non-input consuming (does not require user input). Transitions may comprise one or more conditions and transitions can follow each other directly.
As used herein, "skip condition" means a condition that is verified prior to executing an instructive step (i.e., entering a node, for example of a flow). If the condition is met (e.g. some values have already been set previous in the system), the system will skip executing the instructional step and will continue as if it has already been executed.
As used herein, "small talk" means generally off-topic conversation elements that may be used to respond to user request. For example, if, during a flow devoted to making an airline reservation, a user asks "do you like to fly yourself?" a preprogrammed response (or a random or other selection from a plurality of possible responses) may be invoked which causes a response such as "I never fly myself, because I live in a machine!".
As used herein, "safety net" means a special flow type which is invoked when a user request fails to satisfy any condition (i.e., no matches were found); it is used to ensure that some form of response is made, after which generally the last active flow is resumed where it left off.
As used herein, an "example input" is a text sample that is used to describe, build, and test language recognition rules, triggers, and flows. Example inputs are typically obtained from text sample corpora, which are often obtained for example from logs of actual interactions between users and service representatives such as agents handling chat interactions or phone calls (which can be transcribed to generate example inputs). Typically, an example input (EI) will belong to at least one group of EIs that are related to a specific part of a natural language solution, such as a trigger and a language recognition rule (the trigger's condition). So an EI may "belong to" a specific trigger of a flow, and represents an example of what type of input should fulfill the language recognition rule of the trigger (in other words, the EI is assigned to a specific language recognition rule; often this occurs automatically because the language recognition rules was auto-generated from a set of example inputs, each of which is then assigned as a positive example input for the language recognition rule by default. Each language recognition rule (which can be linked to a trigger, language object, listener, internal flow transition or a skip condition) typically comprises four sets of assigned EIs: positive local EIs (that should match the language recognition rule when tested in isolation, but may not necessarily trigger it when tested in the context of the whole solution); negative local EIs (that should not match it); positive global EIs (that should both match and trigger it); and negative global EIs (that maybe could match it, but should never trigger it). Those sets may be empty. For example, example input "Insurance" may trigger a general language recognition rule; "Vehicle insurance" may fulfill a more specific language recognition rule; and "Car insurance" may trigger the most specific rule of the three. In this situation, the "car insurance" EI will always match the general "insurance" rule; a designer can solve the implied conflict here by setting an ordering relation that always evaluates the "car insurance" rule before the more general "insurance" rule. In that way, the more general rule is triggered only when the more specific condition isn't fulfilled. Alternatively, systems according to the invention may automatically identify conflicts (such as tangles) and propose a new ordering relation that may be reviewed by a developer.
As used herein, a "dialog hand-over" is when a component that is communicating to a user passes communication control to another component.
As used herein, a "host VA" is a virtual assistant that controls a dialog with a user. All communication with users typically goes via a host VA. In order to find a target VA (virtual assistant), the host may turn to a VA broker in a VA Network. The host may communicate with target VAs and VA brokers using a standardized inter-VA communication protocol. The host decides when to initiate a dialog with a target VA, when to forward a user's input to a target VA or a VA broker, and how to handle a response from a target VA or VA broker (e.g. display natural language response received from the target, respond to a request from the target). The host decides when to drop the dialog with a target. A host VA may also be registered in a VAN as a target VA.
As used herein a "target VA" is a virtual assistant that makes its knowledge available to a network or that performs tasks on behalf of users at the request of a network. The target VA needs to be capable of communicating with other VA's and VA brokers using the inter-VA communication protocol. The target VA informs a VA network of its areas of expertise via a VA Broker. A target may be available/registered in multiple networks.
As used herein a "VA broker" assists host VAs (and possibly other VA brokers) in finding suitable target VAs in VA networks. A VA broker may determine the best suitable target VAs based on various criteria, such as but not limited to the user's natural language input, statistical information about VA targets (user satisfaction, usage patterns etc), structured information provided by the host (user preferences, location details, language preferences). A VA broker may have a register of target VAs with their associated capabilities. Alternatively, in some embodiments the broker assembles a list of virtual assistants that are likely to be able to satisfy a user's request that was submitted by the user's host virtual assistant, and send this list of possible virtual assistants back to the host virtual assistant. The host virtual assistant may then select one of the virtual assistants from the list provided by the broker and contact that virtual assistant with the request directly (typically, using contact information such as IP address that is provided by the broker with the list of eligible virtual assistants). In other embodiments, the host may exercise more direct control, selecting a virtual assistant and connecting the host virtual assistant to the selected virtual assistant, or even receiving responses to the user's request from a plurality of virtual assistants and returning either a list of responses (with or without information concerning which virtual assistant provided each response) or a single response selected by the broker.
As used herein an "inter-VA communication protocol" is a standard protocol used by VAs and VA brokers to exchange information. The protocol makes it possible for VAs of different vendors, built using different technologies and targeted at different devices or interfaces, to exchange relevant data in a structured manner.
Hardware Architecture
Generally, the techniques disclosed herein may be implemented on hardware or a combination of software and hardware. For example, they may be implemented in an operating system kernel, in a separate user process, in a library package bound into network applications, on a specially constructed machine, on an application-specific integrated circuit (ASIC), or on a network interface card.
Software/hardware hybrid implementations of at least some of the embodiments disclosed herein may be implemented on a programmable network-resident machine (which should be understood to include intermittently connected network-aware machines) selectively activated or reconfigured by a computer program stored in memory. Such network devices may have multiple network interfaces that may be configured or designed to utilize different types of network communication protocols. A general architecture for some of these machines may be disclosed herein in order to illustrate one or more exemplary means by which a given unit of functionality may be implemented. According to specific embodiments, at least some of the features or functionalities of the various embodiments disclosed herein may be implemented on one or more general-purpose computers associated with one or more networks, such as for example an end-user computer system, a client computer, a network server or other server system, a mobile computing device (e.g., tablet computing device, mobile phone, smartphone, laptop, and the like), a consumer electronic device, a music player, or any other suitable electronic device, router, switch, or the like, or any combination thereof. In at least some embodiments, at least some of the features or functionalities of the various embodiments disclosed herein may be implemented in one or more virtualized computing environments (e.g., network computing clouds, virtual machines hosted on one or more physical computing machines, or the like). Moreover, in some embodiments one or more aspects, or all aspects, of the invention may optionally be implemented via a specially programmed chip (for instance, an application specific integrated circuit, or ASIC, or an erasable programmable read only memory, or EPROM), or via some other hardware-only approach known in the art.
Referring now to
In one embodiment, computing device 100 includes one or more central processing units (CPU) 102, one or more interfaces 110, and one or more busses 106 (such as a peripheral component interconnect (PCI) bus). When acting under the control of appropriate software or firmware, CPU 102 may be responsible for implementing specific functions associated with the functions of a specifically configured computing device or machine. For example, in at least one embodiment, a computing device 100 may be configured or designed to function as a server system utilizing CPU 102, local memory 101 and/or remote memory 120, and interface(s) 110. In at least one embodiment, CPU 102 may be caused to perform one or more of the different types of functions and/or operations under the control of software modules or components, which for example, may include an operating system and any appropriate applications software, drivers, and the like.
CPU 102 may include one or more processors 103 such as, for example, a processor from one of the Intel, ARM, Qualcomm, and AMD families of microprocessors. In some embodiments, processors 103 may include specially designed hardware such as application-specific integrated circuits (ASICs), electrically erasable programmable read-only memories (EEPROMs), field-programmable gate arrays (FPGAs), and so forth, for controlling operations of computing device 100. In a specific embodiment, a local memory 101 (such as non-volatile random access memory (RAM) and/or read-only memory (ROM), including for example one or more levels of cached memory) may also form part of CPU 102. However, there are many different ways in which memory may be coupled to system 100. Memory 101 may be used for a variety of purposes such as, for example, caching and/or storing data, programming instructions, and the like.
As used herein, the term "processor" is not limited merely to those integrated circuits referred to in the art as a processor, a mobile processor, or a microprocessor, but broadly refers to a microcontroller, a microcomputer, a programmable logic controller, an application-specific integrated circuit, and any other programmable circuit.
In one embodiment, interfaces 110 are provided as network interface cards (NICs). Generally, NICs control the sending and receiving of data packets over a computer network; other types of interfaces 110 may for example support other peripherals used with computing device 100. Among the interfaces that may be provided are Ethernet interfaces, frame relay interfaces, cable interfaces, DSL interfaces, token ring interfaces, graphics interfaces, and the like. In addition, various types of interfaces may be provided such as, for example, universal serial bus (USB), Serial, Ethernet, Firewire™, PCI, parallel, radio frequency (RF), Bluetooth™ near-field communications (e.g., using near-field magnetics), 802.11 (WiFi), frame relay, TCP/IP, ISDN, fast Ethernet interfaces, Gigabit Ethernet interfaces, asynchronous transfer mode (ATM) interfaces, high-speed serial interface (HSSI) interfaces, Point of Sale (POS) interfaces, fiber data distributed interfaces (FDDIs), and the like. Generally, such interfaces 110 may include ports appropriate for communication with appropriate media. In some cases, they may also include an independent processor and, in some in stances, volatile and/or non-volatile memory (e.g., RAM).
Although the system shown in
Regardless of network device configuration, the system of the present invention may employ one or more memories or memory modules (such as, for example, remote memory block 120 and local memory 101) configured to store data, program instructions for the general-purpose network operations, or other information relating to the functionality of the embodiments described herein (or any combinations of the above). Program instructions may control execution of or comprise an operating system and/or one or more applications, for example. Memory 120 or memories 101, 120 may also be configured to store data structures, configuration data, encryption data, historical system operations information, or any other specific or generic non-program information described herein.
Because such information and program instructions may be employed to implement one or more systems or methods described herein, at least some network device embodiments may include nontransitory machine-readable storage media, which, for example, may be configured or designed to store program instructions, state information, and the like for performing various operations described herein. Examples of such nontransitory machine-readable storage media include, but are not limited to, magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROM disks; magneto-optical media such as optical disks, and hardware devices that are specially configured to store and perform program instructions, such as read-only memory devices (ROM), flash memory, solid state drives, memristor memory, random access memory (RAM), and the like. Examples of program instructions include both object code, such as may be produced by a compiler, machine code, such as may be produced by an assembler or a linker, byte code, such as may be generated by for example a Java™ compiler and may be executed using a Java virtual machine or equivalent, or files containing higher level code that may be executed by the computer using an interpreter (for example, scripts written in Python, Perl, Ruby, Groovy, or any other scripting language).
In some embodiments, systems according to the present invention may be implemented on a standalone computing system. Referring now to
In some embodiments, systems of the present invention may be implemented on a distributed computing network, such as one having any number of clients and/or servers. Referring now to
In addition, in some embodiments, servers 320 may call external services 370 when needed to obtain additional information, or to refer to additional data concerning a particular call. Communications with external services 370 may take place, for example, via one or more networks 310. In various embodiments, external services 370 may comprise web-enabled services or functionality related to or installed on the hardware device itself. For example, in an embodiment where client applications 230 are implemented on a smartphone or other electronic device, client applications 230 may obtain information stored in a server system 320 in the cloud or on an external service 370 deployed on one or more of a particular enterprise's or user's premises.
In some embodiments of the invention, clients 330 or servers 320 (or both) may make use of one or more specialized services or appliances that may be deployed locally or remotely across one or more networks 310. For example, one or more databases 340 may be used or referred to by one or more embodiments of the invention. It should be understood by one having ordinary skill in the art that databases 340 may be arranged in a wide variety of architectures and using a wide variety of data access and manipulation means. For example, in various embodiments one or more databases 340 may comprise a relational database system using a structured query language (SQL), while others may comprise an alternative data storage technology such as those referred to in the art as "NoSQL" (for example, Hadoop, MapReduce, BigTable, and so forth). In some embodiments variant database architectures such as column-oriented databases, in-memory databases, clustered databases, distributed databases, or even flat file data repositories may be used according to the invention. It will be appreciated by one having ordinary skill in the art that any combination of known or future database technologies may be used as appropriate, unless a specific database technology or a specific arrangement of components is specified for a particular embodiment herein. Moreover, it should be appreciated that the term "database" as used herein may refer to a physical database machine, a cluster of machines acting as a single database system, or a logical database within an overall database management system. Unless a specific meaning is specified for a given use of the term "database", it should be construed to mean any of these senses of the word, all of which are understood as a plain meaning of the term "database" by those having ordinary skill in the art.
Similarly, most embodiments of the invention may make use of one or more security systems 360 and configuration systems 350. Security and configuration management are common information technology (IT) and web functions, and some amount of each are generally associated with any IT or web systems. It should be understood by one having ordinary skill in the art that any configuration or security subsystems known in the art now or in the future may be used in conjunction with embodiments of the invention without limitation, unless a specific security 360 or configuration 350 system or approach is specifically required by the description of any specific embodiment.
In various embodiments, functionality for implementing systems or methods of the present invention may be distributed among any number of client and/or server components. For example, various software modules may be implemented for performing various functions in connection with the present invention, and such modules can be variously implemented to run on server and/or client components.
Conceptual Architecture
In order to formulate one or more appropriate responses to request 410, natural language interaction engine 403 carries out a process that may be broadly described as taking place in three steps. First, natural language interaction engine 403 analyzes 421 request 410 to determine its meaning (usually, by determining the user's intent). Analysis 421 may require preprocessing, for example to convert a spoken request 411 into a textual form suitable for further analysis (although it should be noted that, in some embodiments, audio data obtained from a spoken request 411 could be analyzed directly using one or more speech analytics technologies known in the art). Analysis 421 typically involves parsing request 410 and then analyzing its semantic content to determine one or more possible user intentions that motivated request 410. In many cases this may be straightforward (for instance, a request might be "Please set my alarm for 8 am", and accordingly the user's intent would easily be understood once the sentence was parsed), but in other cases it might be anything but straightforward (for instance, a user might say "Get out of here!" which has several possible intentions, depending on context). Thus it could be that output from analysis 421 could involve a certain amount of uncertainty, for instance in cases where an output of analysis step 421 comprises a list of possible user intents, perhaps with weights indicating their likelihood of being the user's actual intention.
Once analysis 421 is completed, natural language interaction engine 403 may undertake a reasoning step 422 to further elaborate or to better determine user intent. Interaction engine 403 may use advanced linguistic interpretation and business rules to simulate "intelligent thinking", allowing interaction engine 403 to appear to reason like a human and to determine a most appropriate way to respond to a user request 410. During this reasoning process 422, interaction engine 403 may take into account many contextual factors such as date, time, location of user, entities and facts discussed in a current or recent dialog, active flows, local context, global context, information picked up in previous dialogs, data about a user, and the like. In addition, inferences may be drawn based on data about one or more users (for example, when users do "this" they normally also want to do "that"). For example, if a user inquires about "next year's insurance premium", interaction engine 403 may understand context of "next year", but may need to seek clarification about whether the user is asking about home or motor insurance. In general, the goal of reasoning 422 is to reduce ambiguity about what a user meant in request 410, and to apply rules or other logical processing to take into account the extended context in which request 410 was made, in order to facilitate the final step of reacting 423. Once user intent is understood and relevant context has been taken into account, natural language interaction engine 403 may select and execute appropriate actions needed to react 423 to request 410. Suitable reactions may be to give an appropriate verbal, textual, visual, or haptic response, to ask for more information to disambiguate user intent (when for example disambiguation could not be carried out during reasoning step 422), to open a webpage, to play a video, to open another application, to automatically fill in a form, to invoke or update or interact with a third party service, to perform some form of multimodal output, or to execute a transaction for example by updating a database or by integrating with one or more backend or e-commerce systems. These three steps analyze 421, reason 422, and react 423) may all happen seamlessly in milliseconds with interaction engine 403 able to handle thousands of interactions simultaneously, or they may occur over an extended period of time, depending on context and the state of the device being used or its connectivity.
Interaction environment 401 normally comprises a dialog interface 410 and an interaction processing core engine 403. Dialog interface 410 performs functions necessary for media-specific interactions with users. A principal role of dialog interface 410 is to receive input from a user in order that interaction engine core 403 can successfully determine a user's meaning (and the user's intent), and to provide feedback or responses back to the user. For example, when interactions are conducted using speech interfaces, interaction may be either via a phone 414 (for example when a virtual assistant is used in place of a conventional IVR application) or via a dedicated voice application 413 (for example when a user of a mobile application or a web site interacts with a virtual assistant using an application, and no phone is involved). In either case (or indeed in other cases, such as embodiments where other means of exchanging voice communications via audio signals—of which many are known in the art—are used, such as interaction with a building control virtual assistant using an intercom system), it is generally necessary to convert speech input into some form of text, and conversely to convert text output into some form of speech output (although, it should be noted, there are limited means known in the art for analyzing speech signals directly—known in the art as speech analytics technologies—and these could be used directly to infer user meaning and intent). Normally this is done by a voice-to-text conversion layer within dialog interface 410, comprising an automated speech recognition (ASR) engine 411 and a text-to-speech (TTS) engine 412. Many of each of these components are well known in the art, such as those available from Nuance™, Acapela™ Ispeech™, Google™ and other vendors; any suitable ASR and TTS engines may be used according to the invention. In some embodiments, ASR 411 and/or TTS 412 are deployed integrally to system 400 (as shown in
Once input has been received, it is passed as input 404 to interaction engine core 403, where it is reacted to and one or more responses 405 may be generated. When responses 405 are generated and sent to dialog interface 401, they are converted (using TTS 412) to speech form or spoken output, if required, and delivered to a user via either or both of a voice application 413 or a phone 414, or if response 405 is already in textual form it is passed directly to a text-based output channel such as email 415, virtual assistant 416, social media 416, and the like. As a further example illustrating how natural language interpretation can be used to enrich multimodal user interactions, consider a combination mobile phone application that proceeds as follows. First, a user presses a button to initiate the application, which displays or plays a welcome prompt (or both) to the user. The user may say an utterance expressing her intent or goal (for example, "I would like to arrange travel to Berlin tomorrow morning.") The application may then display a selection of radio-button style choices on the screen of the mobile phone, for example comprising "Flights", "Hotel", "Car/Transport", and "Recommended". At substantially the same time, the application may speak a phrase such as, "Where would you like to start to make travel plans? Select a button or tell me what you want to do." The user and the application may, throughout the subsequent interaction, use a blend of spoken, text, and haptic interface conventions to exchange information and satisfy the user's original (and any subsequently arising) intention). Thus it can be seen that many examples of natural language interaction applications envisioned by the inventors may involve multimodal interactions comprising two or more interface modalities (such as speech, text, buttons, images, videos, vibrations, sounds, and so forth).
When input 404 is received by interaction engine core 403, it is generally first handled by an input interpretation module 407, which manages the process of handling inputs 405, making various calls to one or more services of natural language interpreter core (NLI core) 402 (which services and associated processes are described in more detail below), in order to determine one or more meanings of a user's input 404 and to infer therefrom the user's intent. That is, a goal of input interpretation module 407 is to "understand" what a user wants, in order that NLI core 402, in conjunction with input interpretation module 407, can formulate one or more appropriate responses to input 404. Once a response type, or the content of a response, is determined, output generation module 408 is tasked with creating an appropriate response 405, which can then be sent to dialog interface 410 for delivery (in an appropriate medium or via an appropriate channel) to the requesting user. In general, every time input interpretation module 407 receives and processes a user input 404, and every time output generation module 408 formulates a response 405 and sends it to dialog interface 410, log file generation module 409 generates one or more log files 421 in order to capture a history of interactions between users and natural language interaction applications. In addition, more or less detailed information pertaining to processes carried out by NLI core 402 is passed to log file generation module 409 as well, in order that not only what transpired is recorded (that is, the inputs 404 and outputs 405), but also how it was done and why (that is, a history of decisions made, rules and flows executed, variables set, and so forth in NLI core 402 is captured). NLI core 402 may, in order to process user inputs or to create responses, require access to one or more external data sources or external services; such access is provided by one or more connection APIs 406, which manage interactions between NLI core 402 (and possibly also input interpretation module 407 and output generation module 408) and external data sources and services. Such external entities may comprise a plurality of enterprise applications 461 such as customer relationship management (CRM) systems, transaction management systems, trouble ticket tracking systems, inventory management systems, and so forth (it will be appreciated by one having ordinary skill in the art that essentially any enterprise or third party applications may be interacted with using standards-based APIs or other integration technologies, and thus any such enterprise applications known in the art may be used by interaction engine 401 as needed to serve users). Such external entities may also comprise a plurality of third-party cloud-based applications 462, or any other third party services or applications accessible via public or private interfaces (whether standards-based or proprietary), subject only to findability and security constraints.
NLI core 402, when processing requests received (after preprocessing) from input interpretation module 407, uses one or more solution data repositories (or simply "solutions") 430 that it loads as required. Solutions 430 generally comprise various combinations of business rules 431, integrations 432 (that is, configuration data necessary to carry out communications via connect APIs 406 with enterprise applications 461, third party cloud applications 462, and the like), custom objects 433 (such as script objects used to encapsulate frequently used functionality), domain objects 434 (which are collections of ontologies or ontological data sets that represent knowledge about one or more subject domains such as travel, finance, heart health, furniture, and so forth), language recognition rules 435 built up from language objects (which may comprise a large number of formal models of various language elements for one or more languages; language objects will be discussed in detail below), and flows 436. These solution data 430 components in turn may be preloaded as standardized knowledge bases when a natural language interaction application is created, or they may be hand-crafted by one or more natural language interaction (NLI) developer within build environment 440 (or both; it is quite common to start with a prebuilt set of knowledge base 430 components, and then to add new components or extend the added components from within build environment 440) or part of it can be automatically generated according to the embodied invention.
Build environment 440 comprises several components, organized in a preferred embodiment into a single application 441 (sometimes referred to as a "studio"), from which an NLI developer may access client knowledge and data 444, domain libraries 443, and language libraries 442. Build environment 441 typically comprises a graphical coding tool 445 and one or more automated testing tools 446. Thus, in a typical scenario, a developer building a new natural language interaction application such as a virtual assistant would, within a coding tool 445 in build environment 441, load one or more prebuilt language libraries 442 and domain libraries 443, edit these based on available client knowledge and data 444, and then test the resulting knowledge base 430 using one or more automated (or manual) testing tools 446 in order to test the behavior of her new application prior to its use. Once satisfied with a new solution 430 (and with the associated application's behavior), the developer would publish knowledge base 430, which would then be available for NLI core 402 to load and use in handling user requests 404. A key benefit of the build environment 440 approach envisioned by the inventors is that it enables NLI developers who are not necessarily research scientists (such as computational linguists) to rapidly prototype, test, refine, and then tune natural language interaction applications for particular domains or business needs. This makes it possible to move natural language interaction application development from being an activity typical of linguistic experts with technical skills to one practiced commonly by user without technical or linguistic skills and users such as businesses (large and small) and other organizations desiring to use natural language interaction applications in production.
According to a preferred embodiment of the invention, the process just described is part of an overall closed-loop process. The loop is made "closed" by the availability of an analytics framework 450, which can be used to automatically or manually analyze natural language interaction application performance in order to eliminate poorly functioning dialog elements, improve customer satisfaction, and so forth. Analytics framework 450 generally takes as input a plurality of log files 421 generated by log file generation module 409, although in some embodiments analytics framework 450 may take input directly in real time from interaction engine 410, for example by receiving requests 404, responses 405, raw user inputs, and processed outputs to users as they occur directly from the corresponding components of interaction engine 410 (and of course diagnostic information about decision-making and operational processes occurring within interaction engine 410 could similarly be obtained as they occur). Thus the log file-driven architecture shown and discussed herein should be taken as exemplary and not limiting. In various embodiments, analytics framework may comprise one or more of a top response analysis module 457, a dialog browser 451, a category analysis module 452, a log file loader 453, a report generator 454, and a time-based analytics module 456. In any particular embodiment of the invention, some, most, or all of these components may be present within analytics framework 450; moreover, in some embodiments one or more of these components (or other analytical components known in the art such as business intelligence engines, datamarts, and the like), may be implemented as standalone components separate from analytics framework, including for example delivery as web-based or cloud-based services from a third party analytics platform provider. It should be apparent to one having ordinary skill in the art that any combination or architecture of these and other analytical components may be used according to the invention, and that the particular arrangement shown in
Top response analytics module 457 is a software module that computes a ranked list of responses 405 generated by interaction engine 410 to user requests 404; in many embodiments, a corresponding ranked list of requests 404 that led to a particular response 405 may be provided (often by provision of a "drill-down" capability, so that when viewing a ranked list of responses 405, a user may right-click or take some other action to see what requests 404 led to that response 405 most commonly). Responses 405 (and corresponding requests 404) may be ranked according to one of many numerical or qualitative parameters, such as most popular, most used, most often abandoned from, most often viewed, most often complained about, or any other similar ranking metric. Top response analytics module 457 may automatically generate and maintain ranked lists of responses 405, or may do so on request from an analyst or other authorized user (or both). In some embodiments, users may be allowed to specify one or more custom ranking criteria (as long as they are susceptible of measurement based on data received from interaction engine 410); such custom rankings may be conducted on an ad hoc basis, although useful rankings may be added to the last of available standard rankings, and could be added to any automated ranking processes as well, if desired.
Dialog browser 451 is a user interface element that allows an analyst or other authorized user to view (or hear) one or more interactions or dialogs between users and natural language interaction applications. Users may be allowed to request random interaction sessions, in which case one or more sessions would be displayed for viewing at a time; such a view would be useful for example to a marketing analyst who wants to "see" how users interact by randomly browsing among user experiences. Users may also select one or more sessions for viewing (or listening), based typically on any of a wide range of filters or selection criteria made available through a user interface of dialog browser 451. For example, a user could request to view all interactions with a specific user, or with a specific user/application pair, or relating to a particular subject such as a particular product or small talk, and so forth. Users may also request a specific interaction by providing a time, or a session identifier, or a customer name and a time, or any other filtering arrangement that can distinguish the desired session from all others.
Category analysis module 452 is typically a graphical user interface component that allows an analyst or other authorized user to view one or more, or all, sessions or interactions that pertain to a particular category (such as a category within a domain model, for instance by requesting to view a sampling of sessions involving furniture-related questions). Users of category analysis module 452 may be able, for example, to view all or a set of sessions (or to view aggregated numerical data pertaining to such sessions) that were about home equity loans. While viewing such a set of data or sessions, the user might be able to explore for example whether certain questions related to home equity loans are more commonly received from customers that are actually considering taking out such a loan; this might be determined, for example, by displaying columns of data about "probability of loan application being taken" and "probability of closing of loan applications taken", and allowing a user to rank sessions by one or another column in order to understand such patterns. It should be clear to one having ordinary skill in the art of data analytics that there are many ways of viewing and analyzing data of this type, any of which may be included in category analysis module 452 according to the invention.
Time-based analysis module 456 is typically a graphical user interface element that allows an analyst or other authorized user to configure, conduct, schedule, and view the results of various analyses of one or more time-based behaviors of natural language interaction applications and/or their users. For example, time-based analysis module 456 may in some embodiments be used to identify trends in user behavior, such as the emergence of new dominant questions or shifts in linguistic usage over time. Another example, likely to occur quite readily, would be the use of time-based analysis module 456 to study assimilation of new application capabilities upon their introduction in one or more natural language interaction applications. When new capabilities (for instance, new language models, improved business rules, richer interaction modality options, or expansion of domain coverage) are added to an application such as a virtual assistant, their use by those who interact with the modified natural language interaction application is likely to vary over time as more and more users become accustomed to the new capabilities; such user assimilation can be studied to learn how long to wait before expecting changes in user behavior (such knowledge would be quite useful in avoiding for example the premature termination of an experimental feature, which might occur if knowledge engineers misunderstood a normal "uptake curve" for lack of user interest). Similarly, in some embodiments time-based analysis of users' natural language inquiries may uncover emerging linguistic elements, such as the emergence of new "buzzwords", common off-topic questions such as "What do you think of the Avengers movie?", or topics of interest (any of which might suggest modifications to knowledge base to maintain a high degree of user acceptance and virtual assistant fluency).
Report generator 454 is typically a graphical user interface element that allows an analyst or other authorized user to design, configure, schedule, edit, manage distribution of, or view one or more periodic or ad hoc reports regarding operation of interaction engine 410 or virtual assistants hosted by it. Similarly to output from time-based analysis module 456, reports generally prove very useful in detecting trends or emerging problems with natural language interaction applications, language or domain models, or the infrastructure or applications of interaction engine 410. For example, a report might provide tabular and graphical data pertaining to the number of occurrences of various types of interactions using one or more applications, and data for each row or category of interaction for each time period might include items such as the number and/or percentage of sessions that terminated either prematurely or in an unexpected way, the number of sessions that resulted in or involved a product sales inquiry, the number of those that resulted in a sale, and so forth. It is common in the art for reports to be useful in understanding such issues as diurnal patterns of user behavior (for example, users might be more likely to conclude purchases during early evening hours, and might engage in a higher level of off-topic "small talk" in the hours after midnight). It should be clear to one having ordinary skill in the art that any of the many infrastructure, data presentation, drill-down and drill-up, filtering, and other options known in the well-developed art of enterprise application reporting may be used according to the invention. Of course, the range of inferences that might be made through use of analytics based on natural language interaction engine 403 data is quite broad. For example, in some implementations it may be desirable to identify particular channels (media types, such as email, phone, kiosk, instant messaging, and so forth), particular operating systems, particular browsers, and similar specifics regarding a plurality of users. As an exemplary use of such data, it may be that users of Macintosh™ computers from Apple™ might be found to demonstrate certain typical habits and preferences, knowledge of which might allow for effective tailoring of natural language interaction engine 403 to better serve those customers.
Log file loader 453 is typically a graphical user interface element that allows an analyst or other authorized user to manage the configuration and scheduling of log file loading (or to configure real time data acquisition, when that approach is taken to gathering data from interaction engine 410), and to carry out ad hoc log file activities such as log file pruning, exporting of log files to other systems, archiving or configuring archival of log files onto long term storage systems, and so forth.
In most embodiments, the various components of analytics framework 450 are tightly coupled, for instance by allowing an analyst or other user to move directly from one tool or view to another with a simple user interface action such as selecting a button, choosing a tab, or picking a different view from a context-sensitive menu. For example, a user viewing a list of sessions in a category analysis screen 452 might select one or more sessions from a list (for example, by shift-clicking them using a mouse, or using a check button interface element) and then select "browse dialogs" to view the selected sessions in dialog browser 451. It should be appreciated by one having ordinary skill in the art that many styles of linking of related user interface elements and analytic functionality exist in the art, any of which may be used according to the invention.
According to the embodiment, build environment 400 may be used to develop and deploy natural language applications such as application A 511 and application B 512. In some embodiments, natural language interaction engine 520 is a mobile application that is deployed from an application store or downloaded directly from a web site or other source by a user, and that interacts with one or more separate mobile applications 511, 512 using data from one or more solutions 531, 532. In other embodiments, solutions 531 and 532 might be loaded directly into applications 511 and 512, respectively. In yet other embodiments, a solution 531 and a natural language interaction engine 520 may be embedded fully into a standalone mobile application 511; while in other embodiments individual solutions 531, 532 may be deployed either as separate mobile applications 511, 512 that interact with engine 520 or as modules that are downloaded to device 520 and installed as add-ons to engine 520. Build environment 400 will typically comprise exemplary features described above with reference to
Virtual assistant mediator VAM 607 facilitates communication between one or more PAs 602, one or more CAs 608, and/or one or more external services 630. To facilitate communications, VAM 607 uses PA descriptions stored in PA description database 603 to understand the preferences (for example, a preferred external service such as Expedia.com™ for travel arrangements), relationships to other PAs 602 b (for example, virtual assistants of friends and colleagues), preferred CAs 610 (for example, a virtual assistant that relays stock quotes that uses a particular celebrity's voice pattern), and the like. Furthermore, VAM 607 uses CA descriptions stored in CA description database 611 to be informed of what services may be available. Virtual assistant network 601 provides task repository 604 that contains application logic and natural language understanding rules that can be downloaded and/or integrated into one or more PAs 602, one or more CAs 608, 609, 610, or to external services 630 as needed. VAM 607 auto generates natural language recognition rules for each task and adds it to task repository 604 (for example, product names that are not standard language words may require specific natural language recognition rules). Usage process 612 records details on the usage and interaction of virtual assistant network 601 (for example, which PAs connected to which CAs, survey information about a CA or an external service, trust parameters, satisfaction parameters, connectivity issues, and other information), and saved for analytic purposes to usage database 613. Communication between one or more PAs 602, CAs 608, 609, 610, external services interface 606, and/or other system components, is routed via VAM 607. In some embodiments, VA network administrator 614 may manually process a request forwarded by VAM 607. For example, VA network administrator 614 adapts PA 602 natural language recognition rules to user 620's profile (for example, adapting to a specific dialect or difference in standard vocabulary), or enables and customizes one or more external services 630 (for example, access to a subscription-only investment brokerage website). In another embodiment, VA network administrator 614 may enable a human process to provide input to aid in a request by PA 602 where an automated process may not be able. For example, if a web service requires "captcha" input (that is, a type of challenge-response test used in computing as an attempt to ensure that the response is generated by a human).
To prevent session hijacking, that is, to gain unauthorized access to information, services, or components in virtual assistant network 601, PA 602 may disconnect from CA 608 or another PA 602 at any time (for example, by giving a certain command, or based on some alarm or threshold).
In order to facilitate service CA 610 may connect to an external service 630 via one or more external services interface 606. For example, a request from user 620 for PA 602 to order a pizza may access an online mapping service, such as Google Maps™, to determine which pizza vendor may be closest in proximity. In this example, the vendor has a web services interface that may be used by PA 602 to fulfill the request. In another embodiment, a vendor may already have CA 610 that can be used to interface with PA 602 to fulfill the request or a dialog hand-over happens and user 620 directly with CA 610.
In another embodiment, virtual assistant 710 may handle intermittent connectivity and be able to continue processing requests from user 701 that require connection when connectivity resumes. For example, user 701 may request virtual assistant 710 to block off time in her calendar. In this example this request happens when virtual assistant is unable to connect to user 701's calendar. As such, virtual assistant 710 may add the calendar task to task repository 711. When connectivity resumes, virtual assistant 710 sees the task in the task repository and completes the request.
An exemplary dialog between user 701 and virtual assistant 710 is as follows:
- User 701 is in a car interacting with virtual assistant 710 on user device 704 (for example, embedded into the car audio system). User 701 utters, "Can you check flights to Paris for Friday?"
- Virtual assistant 710 may assume that the flight is to commence in London where user 710 resides. Furthermore, based on data in virtual assistant description database 603, virtual assistant 710 may know that user 701 does not prefer to fly low cost. Virtual assistant 710 utters, "I have found ten flights for Friday. When do you need to be there?"
- User 701 interrupts the dialog by uttering, "We will have to continue later I am home now." Virtual assistant may place a time stamp on the dialog and store it in task repository 711.
- At some later point in time, user 701 starts user device 702 (for example, a laptop computer). Virtual assistant 710 utters, "Hi, what about that flight? Should we continue?"/In this example, user 701 agrees.
- Virtual assistant 710 adapts to device 702 that has a larger screen and utters, "So, I had found ten flights for Friday. Do you want me to show you the list?"
- And so on.
In some embodiments, virtual assistant 710 may, for example, be configured with rules as to when to resume a dialog. For example, in the above example, virtual assistant 710 may not have resumed the dialog after the uttered departure date had passed for the intended flight.
In another embodiment, virtual assistant 710 may, for example, simply state that there is an unfinished dialog.
In another embodiment, virtual assistant 710 may, for example, not resume a task based on the capability of user device 704 that may not be capable of enabling the successful outcome of the request.
In another embodiment, virtual assistant may, for example, prioritize unfinished dialogs and present them to user 701 in order of priority or some other desired order.
In another embodiment, virtual assistant 710 may, for example, collect actions that require connections and not revisit those dialogs with user 701 until connectivity is available.
In another embodiment, user 710 may, for example, explicitly tell virtual assistant 710, in natural language, to postpone the task (for example, "let's talk about that later").
In another embodiment, user 710 may, for example, stop the dialogue with virtual assistant 710 (for example, by uttering a command or interaction via an interface on user device 702).
In another embodiment, dialogue between user 701 and virtual assistant 710 may be interrupted by some external factor (for example, an incoming call on mobile device 703)
In another embodiment, dialogue between user 701 and virtual assistant 710 may be interrupted by user 701 being, for example, distracted and not responding to virtual assistant 710.
In another embodiment, dialogue between user 701 and virtual assistant 710 may be interrupted by the connection between the user 710 and virtual assistant network 601 shuts down.
In a preferred embodiment, tasks and/or dialogs can be resumed by user 701 or virtual assistant network 601, for example, when:
- user 701 has finished a secondary task that it switched to, and a postponed task is still active
- when the connectivity between virtual assistant 710 and virtual assistant network 601 has been restored
- if user 701 explicitly requests virtual assistant 710 to resume a task (for example, "let's go back to that booking")
- if user 701 selects a GUI menu, for example, on user device 703 by clicking to resume a task
- when an interruptive factor stops (for example, an incoming call has finished on user device 702), virtual assistant 710 can resume dialogue with user 701
- when a new session starts on the same user device 702 or another user device 703
In some embodiments, tasks might be device dependent and may not be resumed on another device. For example, a task started on user device 702 (i.e. a laptop computer) may not be resumed on user device 704 (i.e. automobile navigation system)
A postponed task can be cancelled for several reasons:
- Task has passed a pre-configured hibernation period (for example, virtual assistant 710 may be configured such that a task can only be active for a certain amount of time)
- Explicitly by user 710 when resumed by a command (for example, user 701 utters, "forget about that")
- User 701 can graphically delete the task via a GUI on user device 703.
Task manager 714 handles user 701's active and postponed tasks to:
- keep tasks in a data structure (for example, a stack, tree, queue or heap). It will be appreciated by one having ordinary skill in the art that a data structure is a particular way of storing and organizing data in a computer so that it can be used efficiently
- set status of tasks (for example, hibernation period, status, time when task was latest active, etc.)
- store tasks with task parameters
- keep information in virtual assistant network 601 so that it is accessible from different user devices 702, 703, 704 to allow for device independent resumption
- make use of methods for postponement of tasks, resumption of tasks, cancellation of tasks, etc.
- hold resumption strategies and interface to device persistence process described earlier that may adapt to the task conditions, for example, a different strategy may be invoked if a task is resumed after a short interruption as opposed to resuming a task after a day or on a different device or modality. For example, with an interruption of a shorter time period, virtual assistant 710 may utter a specific portion of the dialog, for example, "So, what date did you want to fly?" (It will be appreciated by one having ordinary skill in the art that a shorter time period for an interruption would result in a reduced chance of user 701 forgetting where the dialog left off), whereas for an interruption of a longer time period, virtual assistant 710 may utter a more general statement to refresh the mental state of user 701 for the task, for example, "Should we continue with the flight booking to New York?" (It will be appreciated by one having ordinary skill in the art that a longer time period for an interruption would result in a increased chance of user 701 forgetting where the dialog left off if not forgetting the entire dialog altogether)
Task parameters may be obtained in several ways, for example:
- From another dialog flow
- From another user 701 in a dialogue
- From personal information of user 701
Task parameters and data may have conditions or constraints, for example:
- Local or global in scope (for example, may only be used only in current task or can be used in other tasks)
- Life time of a task parameter
- Device dependent task parameters
As further illustrated, a VA broker 1330 may be utilized, such as to facilitate management of a VAN 1310 or to assist in optimal routing of networked operations. For example, a VA broker 1330 may maintain a repository of information related to specific VAs within a network, such as their respective capabilities or knowledge bases, and may use this information to optimally handle requests as needed. As an example, if a host VA is given a request it is unable to fulfill (or, optionally, that it may be capable of fulfilling but to an unacceptable degree, as might be configured to optimize customer satisfaction with assistant interactions), a request complying with the inter-VA communication protocol might be sent to a broker 1330 for assistance. The request may contain a natural language expression and additional structured data. Based on the data in the request and known information about the VAN 1310, the broker 1330 may then find and return information about one or many suitable VA's for fulfillment. The host VA may decide whether to forward the request to any of the suggested target VAs. If so, the host VA communicates with the target VA using the standardized inter-VA protocol. Alternatively, the VA broker may return a list of target VAs to the host VA, the list comprising target VAs that it determines may be likely to be useful in satisfying the request sent by the host VA. The communication between the user and the target VA is done via the host VA. In this preferred embodiment, the host VA exercises full control over selection of a target VA to fulfill the request. In other embodiments, the host VA may evaluate the suitability of a plurality of target VAs and then select a specific target VA and send the relevant information regarding the selected target VA to the host VA. In yet another embodiment, the VA broker may receive responses from a plurality of target VAs to whom it sent a request received from the host VA, and then select from among the received responses that response which is believed to be most suitable for the received request, and send the selected response to the host VA. Again, it will be seen that the host VA controls the actual user interaction. When a host VA receives a list of eligible target VAs from a broker, the host VA decides when to initiate the dialog with a selected target VA (in this case, selected by the host VA, possibly as a result of user interaction), when to forward a user's input, and how to handle a response from a target (example: display natural language response received from the target, respond to a request from the target). The host VA decides when to drop the dialog with a target, based for example on signals from the user, signals from the target or other reasons decided upon by the host VA. In this manner, it can be appreciated that networked virtual assistant operations may be further optimized through the use of a broker to increase efficiency and relevancy, but that a broker need not be required for a VAN to operate (as the VAs may be able to communicate between themselves, as described previously).
A period of time passes (for example, several hours) and in step 804, user 701 connects to virtual assistant 710 via user device 703 (for example, a tablet device). In step 805, virtual assistant 710 identifies an unfinished task in task repository 711 and notifies user 701 that there is an unfinished task. For example, virtual assistant 710 utters, "at which location would you like your dinner party. User 701 may, in step 806, decide to not continue with the unfinished task and utter a new request to virtual assistant 710. Alternatively, user 701 may choose to continue the unfinished task in step 807. If so, virtual assistant 710 interacts with user 701 to discover the remaining details of the request (for example, location and time of dinner party, etc.). Virtual assistant 710 may connect to other PA 621, 622, CA 608, 609, 610, and or external services 630 to complete the request. When completed, the interaction between user 701 and virtual assistant 710 terminates in step 809.
User 930 interfaces to virtual assistant process 910 by means of voice, text, tactile, and/or video input, or a combination of one or more methods, or by some other means to request processing of one or more tasks (herein referred to as, "the task"). For example, a task to book a flight or a task to make dinner reservations. Virtual assistant core 912 understands the task (for example through the use of automatic speech recognition) and determines how best to process the steps to complete the task. In some embodiments, a task may require virtual assistant core 912 to interface to other virtual assistant 808 (for example, a virtual assistant that may be interfaced to expert system with specific or unique capability, such as a virtual assistant for a furniture company with special information on furniture sales information) via virtual assistant network interface 913 and/or external service 830 via external device interface 920 (for example, information from a device connected to the internet, such as a traffic camera) via connected devices interface 911 to process and complete the task. In some embodiments virtual assistant process 910 may not have a suitable connectivity to other elements required to complete the task. For example, in the case where information from an external service 830 via external device interface 920 is required, external service 830 may require internet connectivity to receive the information required to complete a task (for example, images from a traffic camera to determine traffic patterns along a planned route). In another embodiment, a connection to a virtual assistant on a social network of virtual assistants 801 via virtual assistant network interface may be required to complete a task. In the instances where connectivity is not available, virtual assistant core may add a task to task queue 904. In some embodiments, user 930 may indicate a priority for a task using a one or more interfaces to assign a priority (for example, user 930 is given a choice via a graphical user interface, or via voice command, to indicate low, medium or high priority). User priority process 906 associates a priority to a task record in task queue 904. Once a task is added task queue 904, effort score calculator 902 determines what may be required to complete the task. For example, one or more of the following attributes may be stored as parameters of a task to identify effort required to complete a task and assist in determining when and under what conditions tasks may be performed:
- Amount of data that may be required to transmit, if any, for a task. For example, a time estimate on how long a task will take to complete may be calculated by a combination of the amount of data required to transmit and connection speed of network interface 923.
- Communication mode required, if any, for a task. For example, a task may require a synchronous or asynchronous communication (or a combination of both) with external service 830 via connected devices interface 911.
- Type of connectivity required. Network interface 923 may be a local area network (LAN) covering a small geographic area, such as a school or company; a wide area network (WAN) providing communication in a broad geographic area covering national and international locations; personal area networks (PAN) such as a wireless LAN with a very short range (up to, for example, a few meters), for example when communicating with external service 830 such as, a refrigerator or automobile enabled to provide relevant information about itself; internet connectivity for example, to connect to external service 830 via external device interface 920 that is also connected to the internet; intranet connectivity, for example, to connect to external service 830 via external device interface 920 that may provide corporate or retail information services; virtual private networking capability, for example to connect securely to external services 830, and the like.
- Location and time awareness. For example, GPS coordinates, GSM localization, location proximity via Wi-Fi (or other wireless) services, and the like. It will be appreciated by one having ordinary skill in the art that any of the many class of computer program-level services used to provide specific controls for location and time data as control features in computer programs will be used to determine location and time.
- Time zone of user 930 and/or time zone of required resources that may prevent or enable interface to said resource. For example, time zone of other virtual assistant 809, time zone of user 820 who may be the owner of other virtual assistant 809, time zone of external service 830, and the like.
In addition to the above attributes, connectivity process 903 determines a more detailed connectivity requirement for each task to more accurately assist in determining a priority for a task:
- Type of connectivity required for a task. For example, connectivity to a local area network (LAN) covering a small geographic area, such as a school or company where the required service is located; a wide area network (WAN) providing communication in a broad geographic area covering national and international locations; personal area networks (PAN) such as a wireless LAN with a very short range (up to, for example, a few meters); internet connectivity, intranet connectivity, and the like. In some embodiments, one or more types of connectivity may be required
- Signal strength of the current LAN, WAN, PAN, or of internet connection. For example, connectivity process 903 may use signal strength to estimate an amount of time that connectivity may last. Weak signal strength may indicate that a connectivity source is far away and thus an assumption can be made that a connection may only last a few minutes or seconds.
- Signal (for example, a mobile network signal) strength trend. That is, determining, using recent historical metrics, if the signal may be getting stronger or weaker.
- Signal strength (for example, the magnitude of the electric field at a reference point corresponding to virtual assistant process 910 that is a significant distance from the transmitting apparatus) trend. For example, when user 930 is in motion, signal strength may be fluctuating. Using recent historical metrics, signal strength of the apparatus that may be providing connectivity to complete certain tasks for virtual assistant process 910 may be on a progression where strength is increasing or decreasing. Connectivity process 903 may determine that when an increasing signal strength trend is present, connectivity may last longer, whereas on a decreasing signal strength trend, connectivity may not, for example, last long. In some embodiments, signal strength trends may be used in conjunction with location information from location awareness process 921. For example, connectivity process 903 may cross-reference signal strength trend with proximity to, for example, a nearby city where the likelihood of a signal transmitting device is high. In this example, if the signal strength trend is intensifying coupled with location information that user 930 is approaching a large metropolitan area, connectivity process may determine that the likelihood of a long duration of connectivity may be probable. In this example, virtual assistant core 912 may be able to complete high-effort tasks, that is tasks that are deemed to require a larger than typical effort as determined by effort score calculator 902. For example, booking a flight with other virtual assistant 810 that may require a back and forth dialog with user 930. As such, reprioritization process 905 may reprioritize tasks in task queue 904 accordingly.
- Connection type selection. Connectivity process 903, may determine that certain tasks may only be performed on certain types of connections. For example, if a task that may require a large transmittal of data which may, in some connectivity types cost more (for example, on a cellular network where user 930 is roaming), connectivity process 903 may require that a task only use, for example, a lower cost network, such as a Wi-Fi network.
- Privacy of tasks. If a task has privacy requirements, connectivity process 903 may require that a certain type of connection be available on network interface 923. For example, a network capable of supporting a VPN connection.
In a particular embodiment, connectivity process 903 may predict when connectivity may be available next by using location information from location awareness process 921. For example, in the case where user 930 may be currently offline but requesting tasks from virtual assistant process 910 and may be in a state of motion (for example, user 930 may be on a train headed towards a metropolitan area), connectivity process 903, may use location information to determine the amount of time to be within range of known mobile of Wi-Fi networks, or some other form of connectivity, in order to start or continue processing tasks that may have been requested while user virtual assistant process 910 was not connected or continue processing a task that virtual assistant process 910 may have been processing previously and connection lost. In some embodiments, connectivity process 903 may assume that there will be connectivity available, given location information from location awareness process 921, even if it may not known for sure that connection may become available. In a particular embodiment, location awareness process 921 may use other information (for example, speed, direction, and the like) to estimate when connectivity on network interface 923 may be available. In some embodiments, virtual assistant process 910 may notify user 930 of a predicted time-to-connectivity (that is, how long it may be until virtual assistant process 910 may establish a connection to one or more networks required to complete tasks).
Once effort score calculator 902 has determined details and environment that may be required to perform and complete a task, effort score calculator 902 computes an effort score, for example, how long a task may take to complete, what type of connection is required, etc. Effort score calculator 902 may store an effort score for each task in task queue 904. In some embodiments, a duration required to complete a task may be stored for each task in task queue 904. In a particular embodiment, effort score calculator will may be required for a task where a priority was set by user 930 using user priority process 906.
Once a plurality of tasks are stored in task queue 904, reprioritization process 905 may determine that a reprioritization of tasks may be required. For example, if connectivity is intermittent through network interface 923 (for example, connectivity availability is unpredictable and may connect and disconnect at unpredictable periods), reprioritization process 905 may reprioritize tasks so that tasks with lowest effort scores may be attempted first. For example, a task where a length of time to complete is minimal and/or where minimum amounts of data transfer may be required may be prioritized.
In a particular embodiment, user priority process 906 may override reprioritization process 905 with an urgent task. For example, if user 930 requests virtual assistant process 910 to prioritize a task (herein, referred to "priority task") to completion before all other tasks, reprioritization process will keep priority task in the first position, that is, the current task to attempt to complete by virtual assistant process 910. In a particular embodiment, a priority task may be set to a high priority. In this case, user priority process may increase the priority of priority task so that it gets the majority of processing time, while still allowing reprioritization process to process other tasks based on attributes of all tasks in task queue 904 and current environment parameters (for example, time left until next connectivity with a long duration).
In a particular embodiment reprioritization process may prioritize tasks in task queue 904 based on connectivity speeds. For example, if there are tasks with low data requirements, said tasks would be prioritized before tasks that may require a large amount of data to be transferred.
In a preferred embodiment, when a task is not complete and network interface 923 loses connectivity to the required network, the task will be marked as incomplete in task queue 904. When connectivity is re-established on network interface 923, reprioritization process 905 may keep said task as top priority or in some situations, another task may take priority based on environmental parameters. For example, a connection where it is predicted that connectivity may not be active for the duration required for an unfinished task, a task with a low effort score may then be prioritized. The unfinished task may be attempted again when a suitable connection via network interface 923 is available.
In a particular embodiment, virtual assistant core 912 may provide a report to user 930 on the status of tasks in task queue 904. For example, a list may be presented with a list (for example via a graphical user interface such as a screen of a tablet device) of tasks that may have been completed, a percentage completion on tasks that may only be partially complete, and a list of tasks that where processing may not have been attempted.
In a another embodiment, virtual assistant core 912 may present estimations on when tasks can be completed and presented to user 930 (for example by uttering voice signifying as such).
In a preferred embodiment, reprioritization process 905 may be continuously adapting and reprioritizing tasks based on data available about tasks and the current environment (such as the status of connectivity. For example, using predictive elements such as when the next time network interface 923 will have a connection to a required network and types of connectivity available (for example, Wi-Fi, GSM, and the like).
Referring back to step 1101, if it is determined that there is no connectivity on network interface 923, using location data obtained in step 1102, connectivity process 903 may calculate, in step 1103, a predicted time to when network interface 923 may have connectivity. In this example, reprioritization process 905 may reprioritize tasks in task queue 904 based on the predicted connectivity attributes (for example, connectivity process 903 may have predicted that network interface 923 may have connection in five minutes and based on location information from location awareness process 921, connection will stay active for only several minutes. In this example, reprioritization process 905 may reprioritize tasks in task queue 904 as more information becomes available. In step 1104, connectivity predictions are stored in usage database 813 via usage process 812. In a particular embodiment, usage database 813 may reside on a laptop that may also be executing a virtual assistant process 910 or some other portable device). Once connectivity is established on network interface 923, step 1105 compares predictions made in step 1103 to actual data of when connectivity was established, duration of connectivity and any other predictions that were made by connection process 903. As such, prediction algorithms are updated as necessary. In a particular embodiment, historical data will be looked up as needed and used to predict connectivity attributes for said location for future task processing requirements by task process 901.
It will be appreciated by one having ordinary skill in the art that other modes of communication and control are possible. For example, while normally a host VA is in control throughout the process, selecting a target VA from a list provided by a VA broker, controlling the user dialog, and determining precisely how a response received from another VA (i.e., a target VA) is actually presented to the user, and so forth. Alternatively, a VA broker may send to the host VA, instead of a list of target VAs, a list of responses received from a plurality of target VAs, and the host VA can then select the response it prefers. Or, the VA broker can evaluate a set of responses to a request that is received from a plurality of target VAs, and the VA broker makes the selection of the best response, then sending only the selected response to the host VA. Many variations are possible, as should be evident.
It should also be apparent that, among other things, a plurality of virtual assistant networks operating as described herein may in effect provide a "network of knowledge", wherein a user can interact with a host VA by querying it for information, and the host VA can access a potentially very extensive network of VAs, each of which possesses at least a knowledge base, thereby making the entire network of knowledge bases accessible in a natural way to a user via a single virtual assistant user interface.
The skilled person will be aware of a range of possible modifications of the various embodiments described above. Accordingly, the present invention is defined by the claims and their equivalents.
Claims (6)
Priority Applications (3)
|Application Number
|Priority Date
|Filing Date
|US201361769191P true
|US201414184697A true
|US14/242,868 US9172747B2 (en)
|System and methods for virtual assistant networks
Applications Claiming Priority (1)
|Application Number
|Priority Date
|Filing Date
|US14/242,868 US9172747B2 (en)
|System and methods for virtual assistant networks
Related Parent Applications (1)
|Application Number
|Priority Date
|Filing Date
|US201414184697A Continuation
Publications (2)
|Publication Number
|Publication Date
|US20140244712A1 US20140244712A1 (en)
|US9172747B2 true US9172747B2 (en)
Family Applications (1)
|Application Number
|Priority Date
|Filing Date
|US14/242,868 Active US9172747B2 (en)
|System and methods for virtual assistant networks
Country Status (1)
|US9172747B2 (en)
Cited By (10)
|Publication number
|Priority date
|Publication date
|US9519461B2 (en)
|Viv Labs, Inc.
|Dynamically evolving cognitive architecture system based on third-party developers
|US9594542B2 (en)
|Viv Labs, Inc.
|Dynamically evolving cognitive architecture system based on training by third-party developers
|US20170090824A1 (en) *
|International Business Machines Corporation
|Layered queue based coordination of potentially destructive actions in a dispersed storage network memory
|US9633317B2 (en)
|Viv Labs, Inc.
|Dynamically evolving cognitive architecture system based on a natural language intent interpreter
|US20180075335A1 (en) *
|International Business Machines Corporation
|System and method for managing artificial conversational entities enhanced by social knowledge
|WO2019139796A1 (en) *
|Microsoft Technology Licensing, Llc
|Federated intelligent assistance
|US10394957B2 (en)
|Microsoft Technology Licensing, Llc
|Signal analysis in a conversational scheduling assistant computing system
|US10474961B2 (en)
|Viv Labs, Inc.
|Dynamically evolving cognitive architecture system based on prompting for additional user input
|US10673787B2 (en) *
|Servicenow, Inc.
|Virtual agent conversation service
|US10679156B1 (en)
|Wells Fargo Bank, N.A.
|Voice enabled assistant for community demand fulfillment
Families Citing this family (97)
|Publication number
|Priority date
|Publication date
|US9330720B2 (en)
|Apple Inc.
|Methods and apparatus for altering audio output signals
|US10176827B2 (en)
|Verint Americas Inc.
|Active lab
|US20100030549A1 (en)
|Lee Michael M
|Mobile device having human language translation capability with positional feedback
|US8676904B2 (en)
|Apple Inc.
|Electronic devices with voice command and contextual data processing capabilities
|US10489434B2 (en)
|Verint Americas Inc.
|Leveraging concepts with information retrieval techniques and knowledge bases
|US8943094B2 (en)
|Next It Corporation
|Apparatus, system, and method for natural language processing
|US8682667B2 (en)
|Apple Inc.
|User profiling for selecting user specific voice input processing information
|US9122744B2 (en)
|Next It Corporation
|System and method for providing distributed intelligent assistance
|US9262612B2 (en)
|Apple Inc.
|Device access using voice authentication
|US9836177B2 (en)
|Next IT Innovation Labs, LLC
|Providing variable responses in a virtual-assistant environment
|US9223537B2 (en)
|Next It Corporation
|Conversation user interface
|US9536049B2 (en)
|Next It Corporation
|Conversational virtual healthcare assistant
|US20140253455A1 (en) *
|Nuance Communications, Inc.
|Task assistant providing contextual suggestions
|US10445115B2 (en)
|Verint Americas Inc.
|Virtual assistant focused user interfaces
|WO2014197335A1 (en)
|Apple Inc.
|Interpreting and acting upon commands that involve sharing information with remote devices
|EP2881898A1 (en) *
|Accenture Global Services Limited
|Virtual assistant interactivity platform
|US9830044B2 (en)
|Next It Corporation
|Virtual assistant team customization
|JP6390323B2 (en) *
|Vehicle information providing device
|US9430463B2 (en)
|Apple Inc.
|Exemplar-based natural language processing
|US9633004B2 (en)
|Apple Inc.
|Better resolution when referencing to concepts
|EP3480811A1 (en)
|Apple Inc.
|Multi-command single utterance input method
|CN106463110A (en) *
|Information processing device, information processing method, and program
|US20160071517A1 (en)
|Next It Corporation
|Evaluating Conversation Data based on Risk Factors
|US9818400B2 (en)
|Apple Inc.
|Method and apparatus for discovering trending terms in speech requests
|US10642873B2 (en) *
|Microsoft Technology Licensing, Llc
|Dynamic natural language conversation
|US9769227B2 (en)
|Microsoft Technology Licensing, Llc
|Presentation of computing environment on multiple devices
|US10635296B2 (en)
|Microsoft Technology Licensing, Llc
|Partitioned application presentation across devices
|US10448111B2 (en)
|Microsoft Technology Licensing, Llc
|Content projection
|US10074360B2 (en)
|Apple Inc.
|Providing an indication of the suitability of speech recognition
|US9668121B2 (en)
|Apple Inc.
|Social reminders
|US10127911B2 (en)
|Apple Inc.
|Speaker identification and unsupervised speaker adaptation techniques
|US20160100221A1 (en) *
|StreamRail Ltd.
|System and method for preventing abandonment of web-based video content
|US10147421B2 (en) *
|Microcoft Technology Licensing, Llc
|Digital assistant voice input integration
|US10567477B2 (en)
|Apple Inc.
|Virtual assistant continuity
|US9886953B2 (en)
|Apple Inc.
|Virtual assistant activation
|US9721566B2 (en)
|Apple Inc.
|Competing devices responding to voice triggers
|US9578173B2 (en)
|Apple Inc.
|Virtual assistant aided communication with 3rd party service in a communication session
|US10338959B2 (en)
|Microsoft Technology Licensing, Llc
|Task state tracking in systems and services
|US10631057B2 (en)
|Nuance Communications, Inc.
|System and method for natural language driven search and discovery in large data sources
|CN106409295B (en) *
|Method and device for recognizing time information from natural voice information
|US10049668B2 (en)
|Apple Inc.
|Applying neural network language models to weighted finite state transducers for automatic speech recognition
|US10212119B2 (en) *
|Pooja Bisarya
|Method and system for enabling messaging between users
|US20170222956A1 (en) *
|Priya Jain
|Method and system for enabling an auto response facility between users
|US10635281B2 (en)
|Microsoft Technology Licensing, Llc
|Natural language task completion platform authoring for third party experiences
|US10412153B2 (en) *
|Bentley J. Olive
|Methods and systems for facilitating information and expertise distribution via a communications network
|US10133612B2 (en) *
|Nuance Communications, Inc.
|Session processing interaction between two or more virtual assistants
|DK201670540A1 (en)
|Apple Inc
|Application integration with a digital assistant
|US20180018616A1 (en) *
|Avaya Inc.
|Systems and methods for lifecycle management of limited duration knowledge in automated interaction systems
|US9596350B1 (en) *
|Genesys Telecommunications Laboratories, Inc.
|Virtual interactions in contact center operations
|US10115400B2 (en)
|Sonos, Inc.
|Multiple voice services
|US10685656B2 (en)
|Bose Corporation
|Accessing multiple virtual personal assistants (VPA) from a single device
|US10474753B2 (en)
|Apple Inc.
|Language identification using recurrent neural networks
|US20180084392A1 (en) *
|Ebay Inc.
|Text messaging hub system providing access to local and remote service applications
|US10043516B2 (en)
|Apple Inc.
|Intelligent automated assistant
|US9942768B1 (en) *
|International Business Machines Corporation
|Verifying a user utilizing gyroscopic movement
|US10200481B2 (en) *
|The Toronto-Dominion Bank
|System and method for processing an interaction request
|US10437841B2 (en) *
|Microsoft Technology Licensing, Llc
|Digital assistant extension automatic ranking and selection
|US10181323B2 (en)
|Sonos, Inc.
|Arbitration-based voice recognition
|US10529327B1 (en) *
|Parallels International Gmbh
|System and method for enabling voice recognition for operating system
|US10643609B1 (en) *
|Amazon Technologies, Inc.
|Selecting speech inputs
|US10417266B2 (en)
|Apple Inc.
|Context-aware ranking of intelligent response suggestions
|DK201770383A1 (en)
|Apple Inc.
|User interface for correcting recognition errors
|US20180332169A1 (en) *
|Microsoft Technology Licensing, Llc
|Personalization of virtual assistant skills based on user profile information
|US10395654B2 (en)
|Apple Inc.
|Text normalization based on a data-driven learning network
|DK179496B1 (en)
|Apple Inc.
|USER-SPECIFIC Acoustic Models
|KR20190052162A (en) *
|애플 인크.
|Synchronization and task delegation of a digital assistant
|DK201770432A1 (en)
|Apple Inc.
|Hierarchical belief states for digital assistants
|US10303715B2 (en)
|Apple Inc.
|Intelligent automated assistant for media exploration
|US10311144B2 (en)
|Apple Inc.
|Emoji word sense disambiguation
|US10009666B1 (en) *
|Google Llc
|Cross-device handoffs
|US10403278B2 (en)
|Apple Inc.
|Methods and systems for phonetic matching in digital assistant services
|US10657328B2 (en)
|Apple Inc.
|Multi-task recurrent neural network architecture for efficient morphology handling in neural language modeling
|US10657173B2 (en)
|Google Llc
|Validate modification of audio-based computer program output
|US10600409B2 (en)
|Google Llc
|Balance modifications of audio-based computer program output including a chatbot selected based on semantic processing of audio
|US10652170B2 (en)
|Google Llc
|Modification of audio-based computer program output
|US10614122B2 (en)
|Google Llc
|Balance modifications of audio-based computer program output using a placeholder field based on content
|US20180358009A1 (en) *
|International Business Machines Corporation
|Cognitive and interactive sensor based smart home solution
|US20190028587A1 (en) *
|Newvoicemedia, Ltd.
|System and method for integrated virtual assistant-enhanced customer service
|US10445429B2 (en)
|Apple Inc.
|Natural language understanding using vocabularies with compressed serialized tries
|US10621981B2 (en)
|Sonos, Inc.
|Tone interference cancellation
|US10466962B2 (en)
|Sonos, Inc.
|Media playback system with voice assistance
|WO2019074515A1 (en) *
|Hewlett-Packard Development Company, L.P.
|Sub-task assignment based on device capability
|US20190114359A1 (en) *
|Nuance Communications, Inc.
|System And Method For Contextual Reasoning
|US10636424B2 (en)
|Apple Inc.
|Multi-turn canned dialog
|US20190189126A1 (en) *
|Facebook, Inc.
|Methods and systems for responding to inquiries based on social graph information
|EP3502868A1 (en) *
|Harman International Industries, Incorporated
|Intelligent personal assistant interface system
|US10592604B2 (en)
|Apple Inc.
|Inverse text normalization for automatic speech recognition
|US20190303438A1 (en)
|Soundhound, Inc.
|Interpreting Expressions Having Potentially Ambiguous Meanings In Different Domains
|EP3590036A1 (en) *
|Google LLC.
|Determining whether to automatically resume first automated assistant session upon cessation of interrupting second session
|WO2019227099A1 (en) *
|Bpu Holdings Corp.
|Method and system for building artificial and emotional intelligence systems
|DK179822B1 (en)
|Apple Inc.
|Voice interaction at a primary device to access call functionality of a companion device
|DK201870382A1 (en)
|Apple Inc.
|Attention aware virtual assistant dismissal
|US20190373102A1 (en)
|Apple Inc.
|Accelerated task performance
|US20200098372A1 (en)
|Sonos, Inc.
|Voice detection optimization based on selected voice assistant service
|US10692518B2 (en)
|Sonos, Inc.
|Linear filtering for noise-suppressed speech detection via multiple network microphone devices
|KR20200044175A (en) *
|Electronic apparatus and assistant service providing method thereof
|US10586540B1 (en)
|Sonos, Inc.
|Network microphone device with command keyword conditioning
Citations (4)
|Publication number
|Priority date
|Publication date
|US6102969A (en) *
|Netbot, Inc.
|Method and system using information written in a wrapper description language to execute query on a network
|US20120015733A1 (en) *
|Zynga Inc.
|Game mechanics for gifting
|US20130227564A1 (en) *
|Atsushi Asayama
|Information processing apparatus, client management system, and client management method
|US20130304758A1 (en) *
|Apple Inc.
|Crowd Sourcing Information to Fulfill User Requests
- 2014-04-02 US US14/242,868 patent/US9172747B2/en active Active
Patent Citations (4)
|Publication number
|Priority date
|Publication date
|US6102969A (en) *
|Netbot, Inc.
|Method and system using information written in a wrapper description language to execute query on a network
|US20120015733A1 (en) *
|Zynga Inc.
|Game mechanics for gifting
|US20130227564A1 (en) *
|Atsushi Asayama
|Information processing apparatus, client management system, and client management method
|US20130304758A1 (en) *
|Apple Inc.
|Crowd Sourcing Information to Fulfill User Requests
Cited By (12)
|Publication number
|Priority date
|Publication date
|US9519461B2 (en)
|Viv Labs, Inc.
|Dynamically evolving cognitive architecture system based on third-party developers
|US9594542B2 (en)
|Viv Labs, Inc.
|Dynamically evolving cognitive architecture system based on training by third-party developers
|US10474961B2 (en)
|Viv Labs, Inc.
|Dynamically evolving cognitive architecture system based on prompting for additional user input
|US9633317B2 (en)
|Viv Labs, Inc.
|Dynamically evolving cognitive architecture system based on a natural language intent interpreter
|US10083009B2 (en)
|Viv Labs, Inc.
|Dynamically evolving cognitive architecture system planning
|US20170090824A1 (en) *
|International Business Machines Corporation
|Layered queue based coordination of potentially destructive actions in a dispersed storage network memory
|US20180075335A1 (en) *
|International Business Machines Corporation
|System and method for managing artificial conversational entities enhanced by social knowledge
|US10599644B2 (en) *
|International Business Machines Corporation
|System and method for managing artificial conversational entities enhanced by social knowledge
|US10394957B2 (en)
|Microsoft Technology Licensing, Llc
|Signal analysis in a conversational scheduling assistant computing system
|US10673787B2 (en) *
|Servicenow, Inc.
|Virtual agent conversation service
|US10679156B1 (en)
|Wells Fargo Bank, N.A.
|Voice enabled assistant for community demand fulfillment
|WO2019139796A1 (en) *
|Microsoft Technology Licensing, Llc
|Federated intelligent assistance
Also Published As
|Publication number
|Publication date
|US20140244712A1 (en)
Similar Documents
|Publication Date
|JP6498725B2 (en)
|Intelligent automatic assistant
|AU2018204661B2 (en)
|Systems and methods for integrating third party services with a digital assistant
|US10217463B2 (en)
|Hybridized client-server speech recognition
|US10546067B2 (en)
|Platform for creating customizable dialog system engines
|US10482884B1 (en)
|Outcome-oriented dialogs on a speech recognition platform
|RU2637874C2 (en)
|Generation of interactive recommendations for chat information systems
|JP6412914B2 (en)
|Context-dependent processing of interrupts
|US10452783B2 (en)
|Conversational agent
|US10026394B1 (en)
|Managing dialogs on a speech recognition platform
|US10475446B2 (en)
|Using context information to facilitate processing of commands in a virtual assistant
|AU2017204217B2 (en)
|Intelligent automated assistant
|KR102048375B1 (en)
|Using context information to facilitate processing of commands in a virtual assistant
|US10171659B2 (en)
|Customer portal of an intelligent automated agent for a contact center
|US9201865B2 (en)
|Automated assistance for user request that determines semantics by domain, task, and parameter
|US20190180750A1 (en)
|Voice and Connection Platform
|US10170106B2 (en)
|Parameter collection and automatic dialog generation in dialog systems
|CN104704797B (en)
|Virtual protocol communication for electronic equipment
|US10217453B2 (en)
|Virtual assistant configured by selection of wake-up phrase
|CA2917294C (en)
|Intelligent automated agent for a contact center
|US20190214024A1 (en)
|Intelligent automated assistant
|US10079013B2 (en)
|Sharing intents to provide virtual assistance in a multi-person dialog
|US20150045003A1 (en)
|Auto-activating smart responses based on activities from remote devices
|US20150186154A1 (en)
|Virtual assistant team customization
|US8892419B2 (en)
|System and methods for semiautomatic generation and tuning of natural language interaction applications
|US20190327321A1 (en)
|Proactive environment-based chat information system
Legal Events
Owner name: ARTIFICIAL SOLUTIONS IBERIA SL, SPAIN
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LUNDBERG, SONJA PETROVIC;WIEWEG, ANDREAS;HJELM, DAVID;AND OTHERS;SIGNING DATES FROM 20140313 TO 20140328;REEL/FRAME:034155/0058
|Information on status: patent grant
Free format text: PATENTED CASE
Owner name: ARTIFICIAL SOLUTIONS IBERIA S.L., SPAIN
Free format text: CHANGE OF ADDRESS;ASSIGNOR:ARTIFICIAL SOLUTIONS IBERIA S.L.;REEL/FRAME:040577/0996
Effective date: 20161108
|Maintenance fee payment
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YR, SMALL ENTITY (ORIGINAL EVENT CODE: M2551); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY
Year of fee payment: 4