c

bigglue.pipes

MapperPipe

case class MapperPipe[Head <: Identifiable[Head], Input <: Identifiable[Input], Output <: Identifiable[Output], End <: Identifiable[End]](p1: Pipe[Head, Input], mapper: Mapper[Input, Output], p2: Pipe[Output, End]) extends Pipe[Head, End] with Product with Serializable

This is created with the call gitID:--Clone()-->clonedMap (and clonedMap:--CommitExtraction()-->commitInfoMap, but we'll focus on the former) within the example. In the case of gitID:--Clone()-->clonedMap, this is a pipeline that goes from Data Store gitID, computed by the mapper Clone(), and then goes down into Data Store b to be sent further down the pipeline.

Head

The type of the data store that begins the pipeline; bigglue.examples.GitID in this case.

Input

The type of the data store that brings in input to this part of the pipeline; bigglue.examples.GitID In the case of gitID:--Clone()-->clonedMap and bigglue.examples.GitRepo for clonedMap:--CommitExtraction()-->commitInfoMap.

Output

The type of the data store that brings in output to this part of the pipeline; bigglue.examples.GitRepo for gitID:--Clone()-->clonedMap, and bigglue.examples.GitCommitInfo for clonedMap:--CommitExtraction()-->commitInfoMap

End

The type of the data store that shows up at the end of the pipeline; bigglue.examples.GitCommitGroups in this case.

p1

An Input Data Pipe; bigglue.pipes.Implicits.DataNode(gitID) in the example. In the case of clonedMap:--CommitExtraction()-->commitInfoMap, it would be the MapperPipe(gitID:--Clone()-->clonedMap)

mapper

The computation (Mapper) of which the inputs gets computed by. Clone() in the example.

p2

An Output Data Pipe; bigglue.pipes.Implicits.DataNode(clonedMap) in this example. In the case of clonedMap:--CommitExtraction()-->commitInfoMap, it would be bigglue.pipes.Implicits.DataNode(commitInfoMap).

Linear Supertypes
Serializable, Serializable, Product, Equals, Pipe[Head, End], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. MapperPipe
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. Pipe
  7. AnyRef
  8. Any
Implicitly
  1. by any2stringadd
  2. by StringFormat
  3. by Ensuring
  4. by ArrowAssoc
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new MapperPipe(p1: Pipe[Head, Input], mapper: Mapper[Input, Output], p2: Pipe[Output, End])

    p1

    An Input Data Pipe; bigglue.pipes.Implicits.DataNode(gitID) in the example. In the case of clonedMap:--CommitExtraction()-->commitInfoMap, it would be the MapperPipe(gitID:--Clone()-->clonedMap)

    mapper

    The computation (Mapper) of which the inputs gets computed by. Clone() in the example.

    p2

    An Output Data Pipe; bigglue.pipes.Implicits.DataNode(clonedMap) in this example. In the case of clonedMap:--CommitExtraction()-->commitInfoMap, it would be bigglue.pipes.Implicits.DataNode(commitInfoMap).

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. def +(other: String): String
    Implicit
    This member is added by an implicit conversion from MapperPipe[Head, Input, Output, End] to any2stringadd[MapperPipe[Head, Input, Output, End]] performed by method any2stringadd in scala.Predef.
    Definition Classes
    any2stringadd
  4. def ->[B](y: B): (MapperPipe[Head, Input, Output, End], B)
    Implicit
    This member is added by an implicit conversion from MapperPipe[Head, Input, Output, End] to ArrowAssoc[MapperPipe[Head, Input, Output, End]] performed by method ArrowAssoc in scala.Predef.
    Definition Classes
    ArrowAssoc
    Annotations
    @inline()
  5. def :-+[Next <: Identifiable[Next], Mid <: Identifiable[Mid]](headReducer: PartialReducerPipe[End, Mid, Next]): Pipe[Head, Next]

    This connects the Reducer to the input store, connecting it completely.

    This connects the Reducer to the input store, connecting it completely. This is expected to be called after a bigglue.pipes.Implicits.DataNode, and can be seen after gitID and clonedMap.

    Next

    The type at the end of the new pipeline.

    Mid

    The type that the mapper ends up outputting given the input End.

    headReducer

    The reducer to connect to the input store.

    returns

    A connected ReducerPipe that connects this pipeline to the reducer.

    Definition Classes
    Pipe
  6. def :--[Next <: Identifiable[Next], Mid <: Identifiable[Mid]](headMapper: PartialMapperPipe[End, Mid, Next]): Pipe[Head, Next]

    This connects the Mapper to the input store, connecting it completely.

    This connects the Mapper to the input store, connecting it completely. This is expected to be called after a bigglue.pipes.Implicits.DataNode, and can be seen after gitID and clonedMap.

    Next

    The type at the end of the new pipeline.

    Mid

    The type that the mapper ends up outputting given the input End.

    headMapper

    The mapper to connect to the input store.

    returns

    A connected MapperPipe that connects this pipeline to the mapper.

    Definition Classes
    Pipe
  7. def :<[Next <: Identifiable[Next]](parPipes: PartialPipe[End, Next]): Pipe[Head, Next]
    Definition Classes
    Pipe
  8. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  9. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  10. def check(conf: PipeConfig): Unit

    This is called within the example with pipe.check(conf).

    This is called within the example with pipe.check(conf). In basic terms, this checks to see whether the pipeline that we have created is valid.

    conf

    The configuration file that we are checking with.

    Definition Classes
    MapperPipe → Pipe
  11. def clone(): AnyRef
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  12. def end(): DataStore[End]

    This gives you the data store at the end of the pipeline.

    This gives you the data store at the end of the pipeline.

    returns

    The Data Store inside the DataNode at the end of the pipeline. This is the same as p2.end, as that's the part of the pipeline after the Mapper section. In this example, this would be clonedMap for gitID:--Clone()-->clonedMap. Or c for gitID:--Clone()-->clonedMap:--CommitExtraction()-->commitInfoMap.

    Definition Classes
    MapperPipe → Pipe
  13. def ensuring(cond: (MapperPipe[Head, Input, Output, End]) ⇒ Boolean, msg: ⇒ Any): MapperPipe[Head, Input, Output, End]
    Implicit
    This member is added by an implicit conversion from MapperPipe[Head, Input, Output, End] to Ensuring[MapperPipe[Head, Input, Output, End]] performed by method Ensuring in scala.Predef.
    Definition Classes
    Ensuring
  14. def ensuring(cond: (MapperPipe[Head, Input, Output, End]) ⇒ Boolean): MapperPipe[Head, Input, Output, End]
    Implicit
    This member is added by an implicit conversion from MapperPipe[Head, Input, Output, End] to Ensuring[MapperPipe[Head, Input, Output, End]] performed by method Ensuring in scala.Predef.
    Definition Classes
    Ensuring
  15. def ensuring(cond: Boolean, msg: ⇒ Any): MapperPipe[Head, Input, Output, End]
    Implicit
    This member is added by an implicit conversion from MapperPipe[Head, Input, Output, End] to Ensuring[MapperPipe[Head, Input, Output, End]] performed by method Ensuring in scala.Predef.
    Definition Classes
    Ensuring
  16. def ensuring(cond: Boolean): MapperPipe[Head, Input, Output, End]
    Implicit
    This member is added by an implicit conversion from MapperPipe[Head, Input, Output, End] to Ensuring[MapperPipe[Head, Input, Output, End]] performed by method Ensuring in scala.Predef.
    Definition Classes
    Ensuring
  17. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  18. def finalize(): Unit
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  19. def formatted(fmtstr: String): String
    Implicit
    This member is added by an implicit conversion from MapperPipe[Head, Input, Output, End] to StringFormat[MapperPipe[Head, Input, Output, End]] performed by method StringFormat in scala.Predef.
    Definition Classes
    StringFormat
    Annotations
    @inline()
  20. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
  21. def head(): DataStore[Head]

    This gives you the data store at the start of the pipeline.

    This gives you the data store at the start of the pipeline.

    returns

    The Data Store inside the DataNode at the start of the pipeline. This is the same as p1.head, as that's the part of the pipeline before the Mapper section. In this example, it would be gitID.

    Definition Classes
    MapperPipe → Pipe
  22. def init(conf: PipeConfig): Unit

    This is called with the example with pipe.init(conf).

    This is called with the example with pipe.init(conf). This calls mapper.init with conf, and the data stores before and after the mapper computation. This calls p1.end and p2.end for the mapper.init call, p1.init, and p2.init This also moves the init call along the pipeline, sending it to the parts of the pipeline its connected to.

    conf

    The configuration file that we are initializing with. This ideally is the configuration file that is being used to check the pipeline.

    Definition Classes
    MapperPipe → Pipe
  23. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  24. val mapper: Mapper[Input, Output]
  25. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  26. final def notify(): Unit
    Definition Classes
    AnyRef
  27. final def notifyAll(): Unit
    Definition Classes
    AnyRef
  28. val p1: Pipe[Head, Input]
  29. val p2: Pipe[Output, End]
  30. def persist(): Unit

    This is called within the example pipe.persist.

    This is called within the example pipe.persist. This calls mapper.persist, which will check to see what data to send down the pipeline (again?). Then, it will call p1.persist and p2.persist, moving the run call along the pipeline.

    Definition Classes
    MapperPipe → Pipe
  31. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  32. def terminate(): Unit

    This ends the pipeline by terminating the mapper with mapper.terminate.

    This ends the pipeline by terminating the mapper with mapper.terminate. Then, it moves the terminate call throughout the pipeline by calling Pipe.terminate on both p1 and p2.

    Definition Classes
    MapperPipe → Pipe
  33. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  34. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  35. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  36. def ||[OHead <: Identifiable[OHead], OEnd <: Identifiable[OEnd]](o: Pipe[OHead, OEnd]): PairPipes[Head, OHead, End, OEnd]
    Definition Classes
    Pipe
  37. def →[B](y: B): (MapperPipe[Head, Input, Output, End], B)
    Implicit
    This member is added by an implicit conversion from MapperPipe[Head, Input, Output, End] to ArrowAssoc[MapperPipe[Head, Input, Output, End]] performed by method ArrowAssoc in scala.Predef.
    Definition Classes
    ArrowAssoc

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from Pipe[Head, End]

Inherited from AnyRef

Inherited from Any

Inherited by implicit conversion any2stringadd from MapperPipe[Head, Input, Output, End] to any2stringadd[MapperPipe[Head, Input, Output, End]]

Inherited by implicit conversion StringFormat from MapperPipe[Head, Input, Output, End] to StringFormat[MapperPipe[Head, Input, Output, End]]

Inherited by implicit conversion Ensuring from MapperPipe[Head, Input, Output, End] to Ensuring[MapperPipe[Head, Input, Output, End]]

Inherited by implicit conversion ArrowAssoc from MapperPipe[Head, Input, Output, End] to ArrowAssoc[MapperPipe[Head, Input, Output, End]]

Ungrouped