case class ReducerPipe[Head <: Identifiable[Head], Input <: Identifiable[Input], Output <: Identifiable[Output], End <: Identifiable[End]](p1: Pipe[Head, Input], reducer: Reducer[Input, Output], p2: Pipe[Output, End]) extends Pipe[Head, End] with Product with Serializable
In the example, this is created with the call commitInfoMap:-+FindAuthor()+->authorMap within the val pipe = gitID:--Clone()-->clonedMap:--CommitExtraction()-->commitInfoMap:-+FindAuthor()+->authorMap line. With this example, this is a pipeline that goes from Data Store c, computed by the reducer FindAuthor(), and then goes down into Data Store d to be sent further down the pipeline.
- Head
The data store type at the very beginning of the pipeline. In the example, it would be bigglue.examples.GitID
- Input
The type of the Data Store that brings in input to this part of the pipeline. bigglue.examples.GitCommitInfo in this case.
- Output
The type of the Data Store that this part of the pipeline outputs to. bigglue.examples.GitCommitGroups in this case.
- End
The type of the data store that shows up at the end of the pipeline; bigglue.examples.GitCommitGroups in this case.
- p1
The part of the pipeline before the actual reducer section of the pipeline; In the example, it would be the MapperPipe(gitID:--Clone()-->clonedMap:--CommitExtraction()-->commitInfoMap)
- reducer
The computation (Reducer) on which the inputs get computed by. FindAuthor() in the example.
- p2
The part of the pipeline after the actual reducer section of the pipeline; In the example, it would be bigglue.pipes.Implicits.DataNode(authorMap)
- Alphabetic
- By Inheritance
- ReducerPipe
- Serializable
- Serializable
- Product
- Equals
- Pipe
- AnyRef
- Any
- by any2stringadd
- by StringFormat
- by Ensuring
- by ArrowAssoc
- Hide All
- Show All
- Public
- All
Instance Constructors
-
new
ReducerPipe(p1: Pipe[Head, Input], reducer: Reducer[Input, Output], p2: Pipe[Output, End])
- p1
The part of the pipeline before the actual reducer section of the pipeline; In the example, it would be the MapperPipe(gitID:--Clone()-->clonedMap:--CommitExtraction()-->commitInfoMap)
- reducer
The computation (Reducer) on which the inputs get computed by. FindAuthor() in the example.
- p2
The part of the pipeline after the actual reducer section of the pipeline; In the example, it would be bigglue.pipes.Implicits.DataNode(authorMap)
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
def
+(other: String): String
- Implicit
- This member is added by an implicit conversion from ReducerPipe[Head, Input, Output, End] to any2stringadd[ReducerPipe[Head, Input, Output, End]] performed by method any2stringadd in scala.Predef.
- Definition Classes
- any2stringadd
-
def
->[B](y: B): (ReducerPipe[Head, Input, Output, End], B)
- Implicit
- This member is added by an implicit conversion from ReducerPipe[Head, Input, Output, End] to ArrowAssoc[ReducerPipe[Head, Input, Output, End]] performed by method ArrowAssoc in scala.Predef.
- Definition Classes
- ArrowAssoc
- Annotations
- @inline()
-
def
:-+[Next <: Identifiable[Next], Mid <: Identifiable[Mid]](headReducer: PartialReducerPipe[End, Mid, Next]): Pipe[Head, Next]
This connects the Reducer to the input store, connecting it completely.
This connects the Reducer to the input store, connecting it completely. This is expected to be called after a bigglue.pipes.Implicits.DataNode, and can be seen after gitID and clonedMap.
- Next
The type at the end of the new pipeline.
- Mid
The type that the mapper ends up outputting given the input End.
- headReducer
The reducer to connect to the input store.
- returns
A connected ReducerPipe that connects this pipeline to the reducer.
- Definition Classes
- Pipe
-
def
:--[Next <: Identifiable[Next], Mid <: Identifiable[Mid]](headMapper: PartialMapperPipe[End, Mid, Next]): Pipe[Head, Next]
This connects the Mapper to the input store, connecting it completely.
This connects the Mapper to the input store, connecting it completely. This is expected to be called after a bigglue.pipes.Implicits.DataNode, and can be seen after gitID and clonedMap.
- Next
The type at the end of the new pipeline.
- Mid
The type that the mapper ends up outputting given the input End.
- headMapper
The mapper to connect to the input store.
- returns
A connected MapperPipe that connects this pipeline to the mapper.
- Definition Classes
- Pipe
-
def
:<[Next <: Identifiable[Next]](parPipes: PartialPipe[End, Next]): Pipe[Head, Next]
- Definition Classes
- Pipe
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
check(conf: PipeConfig): Unit
This is called within the example with pipe.check(conf).
This is called within the example with pipe.check(conf). In basic terms, this checks to see whether the pipeline that we have created is valid.
- conf
The configuration file that we are checking with.
- Definition Classes
- ReducerPipe → Pipe
-
def
clone(): AnyRef
- Attributes
- protected[java.lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
def
end(): DataStore[End]
This gives you the data store at the end of the pipeline.
This gives you the data store at the end of the pipeline.
- returns
The Data Store inside the DataNode at the end of the pipeline. This is the same as p2.end, as that's the part of the pipeline after the Reducer section. In this example, this would be d.
- Definition Classes
- ReducerPipe → Pipe
-
def
ensuring(cond: (ReducerPipe[Head, Input, Output, End]) ⇒ Boolean, msg: ⇒ Any): ReducerPipe[Head, Input, Output, End]
- Implicit
- This member is added by an implicit conversion from ReducerPipe[Head, Input, Output, End] to Ensuring[ReducerPipe[Head, Input, Output, End]] performed by method Ensuring in scala.Predef.
- Definition Classes
- Ensuring
-
def
ensuring(cond: (ReducerPipe[Head, Input, Output, End]) ⇒ Boolean): ReducerPipe[Head, Input, Output, End]
- Implicit
- This member is added by an implicit conversion from ReducerPipe[Head, Input, Output, End] to Ensuring[ReducerPipe[Head, Input, Output, End]] performed by method Ensuring in scala.Predef.
- Definition Classes
- Ensuring
-
def
ensuring(cond: Boolean, msg: ⇒ Any): ReducerPipe[Head, Input, Output, End]
- Implicit
- This member is added by an implicit conversion from ReducerPipe[Head, Input, Output, End] to Ensuring[ReducerPipe[Head, Input, Output, End]] performed by method Ensuring in scala.Predef.
- Definition Classes
- Ensuring
-
def
ensuring(cond: Boolean): ReducerPipe[Head, Input, Output, End]
- Implicit
- This member is added by an implicit conversion from ReducerPipe[Head, Input, Output, End] to Ensuring[ReducerPipe[Head, Input, Output, End]] performed by method Ensuring in scala.Predef.
- Definition Classes
- Ensuring
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
finalize(): Unit
- Attributes
- protected[java.lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
def
formatted(fmtstr: String): String
- Implicit
- This member is added by an implicit conversion from ReducerPipe[Head, Input, Output, End] to StringFormat[ReducerPipe[Head, Input, Output, End]] performed by method StringFormat in scala.Predef.
- Definition Classes
- StringFormat
- Annotations
- @inline()
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
-
def
head(): DataStore[Head]
This gives you the data store at the start of the pipeline.
This gives you the data store at the start of the pipeline.
- returns
The Data Store inside the DataNode at the start of the pipeline. This is the same as p1.head, as that's the part of the pipeline before the Reducer section. In this example, it would be gitID.
- Definition Classes
- ReducerPipe → Pipe
-
def
init(conf: PipeConfig): Unit
This is called with the example with pipe.init(conf).
This is called with the example with pipe.init(conf). This calls reducer.init with conf, and the data stores before and after the mapper computation. This calls p1.end and p2.end for the reducer.init call, p1.init, and p2.init This also moves the init call along the pipeline, sending it to the parts of the pipeline its connected to.
- conf
The configuration file that we are initializing with. This ideally is the configuration file that is being used to check the pipeline.
- Definition Classes
- ReducerPipe → Pipe
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- val p1: Pipe[Head, Input]
- val p2: Pipe[Output, End]
-
def
persist(): Unit
This is called within the example pipe.persist.
This is called within the example pipe.persist. This calls reducer.persist, which will check to see what data to send down the pipeline (again?). Then, it will call p1.persist and p2.persist, moving the run call along the pipeline.
- Definition Classes
- ReducerPipe → Pipe
- val reducer: Reducer[Input, Output]
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
terminate(): Unit
This ends the pipeline by terminating the reducer with reducer.terminate.
This ends the pipeline by terminating the reducer with reducer.terminate. Then, it moves the terminate call throughout the pipeline by calling Pipe.terminate on both p1 and p2.
- Definition Classes
- ReducerPipe → Pipe
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
def
||[OHead <: Identifiable[OHead], OEnd <: Identifiable[OEnd]](o: Pipe[OHead, OEnd]): PairPipes[Head, OHead, End, OEnd]
- Definition Classes
- Pipe
-
def
→[B](y: B): (ReducerPipe[Head, Input, Output, End], B)
- Implicit
- This member is added by an implicit conversion from ReducerPipe[Head, Input, Output, End] to ArrowAssoc[ReducerPipe[Head, Input, Output, End]] performed by method ArrowAssoc in scala.Predef.
- Definition Classes
- ArrowAssoc